[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251215 21:05:46.906821 23617 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.16.126:42903
I20251215 21:05:46.907881 23617 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:05:46.908496 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:05:46.914659 23625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:46.914721 23623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:46.914742 23622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:05:46.915036 23617 server_base.cc:1047] running on GCE node
I20251215 21:05:46.915458 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:05:46.915552 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:05:46.915591 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832746915589 us; error 0 us; skew 500 ppm
I20251215 21:05:46.917223 23617 webserver.cc:492] Webserver started at http://127.23.16.126:43299/ using document root <none> and password file <none>
I20251215 21:05:46.917729 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:05:46.917783 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:05:46.917987 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:05:46.919512 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/master-0-root/instance:
uuid: "656f090a71e9497b9a25585b2f7091a8"
format_stamp: "Formatted at 2025-12-15 21:05:46 on dist-test-slave-bslw"
I20251215 21:05:46.922802 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251215 21:05:46.924770 23630 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:46.925810 23617 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:05:46.925900 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/master-0-root
uuid: "656f090a71e9497b9a25585b2f7091a8"
format_stamp: "Formatted at 2025-12-15 21:05:46 on dist-test-slave-bslw"
I20251215 21:05:46.925984 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:05:46.962069 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:05:46.962713 23617 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:05:46.962868 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:05:46.963670 23636 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20251215 21:05:46.969431 23682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.126:42903 every 8 connection(s)
I20251215 21:05:46.969435 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.126:42903
I20251215 21:05:46.972034 23683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:05:46.976935 23683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8: Bootstrap starting.
I20251215 21:05:46.979241 23683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8: Neither blocks nor log segments found. Creating new log.
I20251215 21:05:46.979818 23683 log.cc:826] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8: Log is configured to *not* fsync() on all Append() calls
I20251215 21:05:46.981393 23683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8: No bootstrap required, opened a new log
I20251215 21:05:46.983773 23683 raft_consensus.cc:359] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f090a71e9497b9a25585b2f7091a8" member_type: VOTER }
I20251215 21:05:46.983925 23683 raft_consensus.cc:385] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:05:46.983973 23683 raft_consensus.cc:740] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656f090a71e9497b9a25585b2f7091a8, State: Initialized, Role: FOLLOWER
I20251215 21:05:46.984354 23683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f090a71e9497b9a25585b2f7091a8" member_type: VOTER }
I20251215 21:05:46.984467 23683 raft_consensus.cc:399] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:05:46.984521 23683 raft_consensus.cc:493] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:05:46.984602 23683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:05:46.985389 23683 raft_consensus.cc:515] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f090a71e9497b9a25585b2f7091a8" member_type: VOTER }
I20251215 21:05:46.985667 23683 leader_election.cc:304] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656f090a71e9497b9a25585b2f7091a8; no voters:
I20251215 21:05:46.985920 23683 leader_election.cc:290] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:05:46.986050 23686 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:05:46.986258 23686 raft_consensus.cc:697] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 1 LEADER]: Becoming Leader. State: Replica: 656f090a71e9497b9a25585b2f7091a8, State: Running, Role: LEADER
I20251215 21:05:46.986527 23686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f090a71e9497b9a25585b2f7091a8" member_type: VOTER }
I20251215 21:05:46.986867 23683 sys_catalog.cc:565] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:05:46.987911 23688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656f090a71e9497b9a25585b2f7091a8. Latest consensus state: current_term: 1 leader_uuid: "656f090a71e9497b9a25585b2f7091a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f090a71e9497b9a25585b2f7091a8" member_type: VOTER } }
I20251215 21:05:46.988013 23688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [sys.catalog]: This master's current role is: LEADER
I20251215 21:05:46.988534 23695 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:05:46.988556 23687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656f090a71e9497b9a25585b2f7091a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f090a71e9497b9a25585b2f7091a8" member_type: VOTER } }
I20251215 21:05:46.988636 23687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [sys.catalog]: This master's current role is: LEADER
I20251215 21:05:46.991219 23695 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:05:46.991552 23617 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 21:05:46.996227 23695 catalog_manager.cc:1357] Generated new cluster ID: 6a79109a2fca4492b1f7718504ed1a14
I20251215 21:05:46.996326 23695 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:05:47.008132 23695 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:05:47.009174 23695 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:05:47.016650 23695 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8: Generated new TSK 0
I20251215 21:05:47.017308 23695 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:05:47.024214 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:05:47.026629 23706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:47.026683 23705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:47.026825 23708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:05:47.026889 23617 server_base.cc:1047] running on GCE node
I20251215 21:05:47.027073 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:05:47.027115 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:05:47.027138 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832747027137 us; error 0 us; skew 500 ppm
I20251215 21:05:47.028049 23617 webserver.cc:492] Webserver started at http://127.23.16.65:36595/ using document root <none> and password file <none>
I20251215 21:05:47.028193 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:05:47.028242 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:05:47.028316 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:05:47.028673 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-0-root/instance:
uuid: "bf42b6838bde4b4199f10f61556786ef"
format_stamp: "Formatted at 2025-12-15 21:05:47 on dist-test-slave-bslw"
I20251215 21:05:47.030120 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:05:47.030983 23713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.031213 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:05:47.031270 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "bf42b6838bde4b4199f10f61556786ef"
format_stamp: "Formatted at 2025-12-15 21:05:47 on dist-test-slave-bslw"
I20251215 21:05:47.031330 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:05:47.038933 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:05:47.039260 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:05:47.039603 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:05:47.040244 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:05:47.040285 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.040320 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:05:47.040349 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.045941 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:46839
I20251215 21:05:47.045984 23776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:46839 every 8 connection(s)
I20251215 21:05:47.046808 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:05:47.048982 23784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:05:47.049153 23617 server_base.cc:1047] running on GCE node
W20251215 21:05:47.049731 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:47.049919 23781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:05:47.050110 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:05:47.050148 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:05:47.050168 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832747050167 us; error 0 us; skew 500 ppm
I20251215 21:05:47.050928 23617 webserver.cc:492] Webserver started at http://127.23.16.66:44877/ using document root <none> and password file <none>
I20251215 21:05:47.051069 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:05:47.051122 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:05:47.051218 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:05:47.051558 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-1-root/instance:
uuid: "ec6828f7b4fb41348830bc708ea09c2b"
format_stamp: "Formatted at 2025-12-15 21:05:47 on dist-test-slave-bslw"
I20251215 21:05:47.052958 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:05:47.053822 23789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.054085 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:05:47.054145 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-1-root
uuid: "ec6828f7b4fb41348830bc708ea09c2b"
format_stamp: "Formatted at 2025-12-15 21:05:47 on dist-test-slave-bslw"
I20251215 21:05:47.054200 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:05:47.055320 23777 heartbeater.cc:344] Connected to a master server at 127.23.16.126:42903
I20251215 21:05:47.055482 23777 heartbeater.cc:461] Registering TS with master...
I20251215 21:05:47.055794 23777 heartbeater.cc:507] Master 127.23.16.126:42903 requested a full tablet report, sending...
I20251215 21:05:47.057228 23647 ts_manager.cc:194] Registered new tserver with Master: bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:47.058566 23647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45206
I20251215 21:05:47.076402 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:05:47.076894 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:05:47.077209 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:05:47.077778 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:05:47.077828 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.077880 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:05:47.077906 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.083722 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.66:44057
I20251215 21:05:47.083750 23852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.66:44057 every 8 connection(s)
I20251215 21:05:47.084652 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:05:47.086418 23857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:47.087350 23859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:05:47.087414 23856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:05:47.088886 23853 heartbeater.cc:344] Connected to a master server at 127.23.16.126:42903
I20251215 21:05:47.088969 23853 heartbeater.cc:461] Registering TS with master...
I20251215 21:05:47.089164 23853 heartbeater.cc:507] Master 127.23.16.126:42903 requested a full tablet report, sending...
I20251215 21:05:47.089758 23647 ts_manager.cc:194] Registered new tserver with Master: ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:47.090276 23647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45222
I20251215 21:05:47.093586 23617 server_base.cc:1047] running on GCE node
I20251215 21:05:47.093775 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:05:47.093813 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:05:47.093827 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832747093827 us; error 0 us; skew 500 ppm
I20251215 21:05:47.094676 23617 webserver.cc:492] Webserver started at http://127.23.16.67:36119/ using document root <none> and password file <none>
I20251215 21:05:47.094815 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:05:47.094861 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:05:47.094930 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:05:47.095309 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-2-root/instance:
uuid: "31ff5cfb5d624764ae78ac08450e9022"
format_stamp: "Formatted at 2025-12-15 21:05:47 on dist-test-slave-bslw"
I20251215 21:05:47.096702 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:05:47.097669 23864 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.097952 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:05:47.098049 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-2-root
uuid: "31ff5cfb5d624764ae78ac08450e9022"
format_stamp: "Formatted at 2025-12-15 21:05:47 on dist-test-slave-bslw"
I20251215 21:05:47.098126 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:05:47.109414 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:05:47.109779 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:05:47.110050 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:05:47.110468 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:05:47.110503 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.110544 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:05:47.110571 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:05:47.115595 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.67:41817
I20251215 21:05:47.115635 23927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.67:41817 every 8 connection(s)
I20251215 21:05:47.120245 23928 heartbeater.cc:344] Connected to a master server at 127.23.16.126:42903
I20251215 21:05:47.120354 23928 heartbeater.cc:461] Registering TS with master...
I20251215 21:05:47.120558 23928 heartbeater.cc:507] Master 127.23.16.126:42903 requested a full tablet report, sending...
I20251215 21:05:47.121176 23647 ts_manager.cc:194] Registered new tserver with Master: 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817)
I20251215 21:05:47.121757 23617 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005776835s
I20251215 21:05:47.121891 23647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45226
I20251215 21:05:47.134963 23617 test_util.cc:276] Using random seed: -106942481
I20251215 21:05:47.144666 23647 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45256:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:05:47.145838 23647 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:05:47.160746 23741 tablet_service.cc:1505] Processing CreateTablet for tablet fee89533731c48b8979083516919521b (DEFAULT_TABLE table=test-workload [id=cb468633f8ea4bc48a30372d67b2deda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:05:47.160825 23892 tablet_service.cc:1505] Processing CreateTablet for tablet fee89533731c48b8979083516919521b (DEFAULT_TABLE table=test-workload [id=cb468633f8ea4bc48a30372d67b2deda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:05:47.161006 23817 tablet_service.cc:1505] Processing CreateTablet for tablet fee89533731c48b8979083516919521b (DEFAULT_TABLE table=test-workload [id=cb468633f8ea4bc48a30372d67b2deda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:05:47.161203 23817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee89533731c48b8979083516919521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:05:47.161186 23741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee89533731c48b8979083516919521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:05:47.161185 23892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee89533731c48b8979083516919521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:05:47.164918 23951 tablet_bootstrap.cc:492] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Bootstrap starting.
I20251215 21:05:47.165210 23949 tablet_bootstrap.cc:492] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Bootstrap starting.
I20251215 21:05:47.165997 23949 tablet_bootstrap.cc:654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Neither blocks nor log segments found. Creating new log.
I20251215 21:05:47.166002 23951 tablet_bootstrap.cc:654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Neither blocks nor log segments found. Creating new log.
I20251215 21:05:47.166934 23950 tablet_bootstrap.cc:492] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Bootstrap starting.
I20251215 21:05:47.167166 23951 tablet_bootstrap.cc:492] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: No bootstrap required, opened a new log
I20251215 21:05:47.167186 23949 tablet_bootstrap.cc:492] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: No bootstrap required, opened a new log
I20251215 21:05:47.167315 23951 ts_tablet_manager.cc:1403] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:05:47.167414 23949 ts_tablet_manager.cc:1403] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:05:47.167789 23951 raft_consensus.cc:359] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.167937 23951 raft_consensus.cc:385] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:05:47.167893 23950 tablet_bootstrap.cc:654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Neither blocks nor log segments found. Creating new log.
I20251215 21:05:47.167887 23949 raft_consensus.cc:359] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.168006 23951 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Initialized, Role: FOLLOWER
I20251215 21:05:47.168679 23951 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.169008 23950 tablet_bootstrap.cc:492] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: No bootstrap required, opened a new log
I20251215 21:05:47.169091 23951 ts_tablet_manager.cc:1434] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:05:47.169122 23950 ts_tablet_manager.cc:1403] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:05:47.169322 23928 heartbeater.cc:499] Master 127.23.16.126:42903 was elected leader, sending a full tablet report...
I20251215 21:05:47.169318 23949 raft_consensus.cc:385] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:05:47.169409 23949 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Initialized, Role: FOLLOWER
I20251215 21:05:47.169543 23949 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.169874 23949 ts_tablet_manager.cc:1434] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:05:47.170018 23853 heartbeater.cc:499] Master 127.23.16.126:42903 was elected leader, sending a full tablet report...
I20251215 21:05:47.170274 23950 raft_consensus.cc:359] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.170377 23950 raft_consensus.cc:385] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:05:47.170408 23950 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Initialized, Role: FOLLOWER
I20251215 21:05:47.170503 23950 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.170790 23950 ts_tablet_manager.cc:1434] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251215 21:05:47.170912 23777 heartbeater.cc:499] Master 127.23.16.126:42903 was elected leader, sending a full tablet report...
I20251215 21:05:47.171613 23958 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:05:47.171686 23958 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.173907 23958 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:47.175755 23957 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:05:47.175899 23957 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.177721 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef" is_pre_election: true
I20251215 21:05:47.177873 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 0.
I20251215 21:05:47.178107 23957 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:47.178465 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters:
I20251215 21:05:47.178623 23958 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:05:47.178670 23958 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:47.178709 23958 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:05:47.179651 23958 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.179949 23958 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 1 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:47.181037 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:47.181126 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022" is_pre_election: true
I20251215 21:05:47.181205 23827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b" is_pre_election: true
I20251215 21:05:47.181286 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 1: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:47.181365 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:47.181293 23827 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 0.
I20251215 21:05:47.181157 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:05:47.182494 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 1.
I20251215 21:05:47.182765 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:47.182935 23958 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:05:47.183019 23958 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:47.183152 23958 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.184744 23827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b" is_pre_election: true
I20251215 21:05:47.184885 23827 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 0.
I20251215 21:05:47.185137 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:47.185168 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 1 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:47.185375 23960 raft_consensus.cc:2764] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251215 21:05:47.203161 23827 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:05:47.203215 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 1 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:05:47.203605 23958 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:05:47.204223 23827 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 1 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:05:47.204573 23958 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:05:47.250466 23617 test_util.cc:276] Using random seed: -106826977
I20251215 21:05:47.258076 23617 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20251215 21:05:47.584702 23956 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:47.584858 23956 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:05:47.585814 23956 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.586304 23956 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 2 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:47.589475 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:47.589481 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:47.589849 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:47.590029 23956 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 21:05:47.593417 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 2 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 1. Current term is 2. Ops: []
I20251215 21:05:47.593755 23958 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 1.39, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
W20251215 21:05:47.593940 23986 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20251215 21:05:47.593987 23986 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20251215 21:05:47.665210 23986 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 21:05:47.665330 23986 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:47.665527 23986 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.665825 23986 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:05:47.691426 23956 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:47.691543 23956 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:05:47.692392 23956 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.692776 23956 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 3 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:47.693179 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:47.693243 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:47.693383 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:05:47.694195 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 3.
I20251215 21:05:47.694470 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:47.694613 23956 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 3 FOLLOWER]: Leader election won for term 3
I20251215 21:05:47.694751 23956 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 3 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:47.694888 23956 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.696344 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 1 to 3, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 3 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:47.749155 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:47.749282 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:05:47.750176 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.750566 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 2 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:47.751005 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:47.751005 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:47.751199 23902 raft_consensus.cc:2368] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for earlier term 2. Current term is 3.
I20251215 21:05:47.751456 23716 leader_election.cc:400] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 2 election: Vote denied by peer 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817) with higher term. Message: Invalid argument: T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for earlier term 2. Current term is 3.
I20251215 21:05:47.751497 23716 leader_election.cc:403] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251215 21:05:47.751601 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:05:47.752283 24001 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817) with higher term. Message: Invalid argument: T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for earlier term 2. Current term is 3.
I20251215 21:05:47.781899 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251215 21:05:47.782303 23956 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251215 21:05:47.791385 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 3 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251215 21:05:47.791704 24003 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251215 21:05:47.892894 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 3 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:47.893025 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 3 FOLLOWER]: Advancing to term 4
I20251215 21:05:47.893890 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:47.894259 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 4 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:47.894730 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:47.894730 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:47.895156 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:47.895350 24001 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 21:05:47.915692 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 4 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 3. Current term is 4. Ops: []
I20251215 21:05:47.916070 23956 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251215 21:05:48.014181 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 4 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 3. Current term is 4. Ops: []
I20251215 21:05:48.014562 24003 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251215 21:05:48.032706 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:48.032836 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 4 FOLLOWER]: Advancing to term 5
I20251215 21:05:48.033701 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.034068 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 5 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:48.034463 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 5 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:48.034482 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 5 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.034821 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:48.034976 24001 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251215 21:05:48.049456 23956 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 3 LEADER]: Stepping down as leader of term 3
I20251215 21:05:48.049548 23956 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:48.049716 23956 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.049876 23956 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 3 FOLLOWER]: Advancing to term 4
I20251215 21:05:48.139496 23986 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:48.139637 23986 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 3 FOLLOWER]: Advancing to term 4
I20251215 21:05:48.140583 23986 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.141048 23986 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 4 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:48.141548 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.141602 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:48.141748 23751 raft_consensus.cc:2368] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for earlier term 4. Current term is 5.
I20251215 21:05:48.142038 23866 leader_election.cc:400] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 4 election: Vote denied by peer bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839) with higher term. Message: Invalid argument: T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for earlier term 4. Current term is 5.
I20251215 21:05:48.142081 23866 leader_election.cc:403] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251215 21:05:48.142200 23986 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 4 FOLLOWER]: Advancing to term 5
I20251215 21:05:48.142545 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 4.
I20251215 21:05:48.143023 23986 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839) with higher term. Message: Invalid argument: T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for earlier term 4. Current term is 5.
I20251215 21:05:48.257496 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:48.257627 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 5 FOLLOWER]: Advancing to term 6
I20251215 21:05:48.258543 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.259044 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 6 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:48.259445 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.259452 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:48.259558 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 4 FOLLOWER]: Advancing to term 6
I20251215 21:05:48.259584 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 5 FOLLOWER]: Advancing to term 6
I20251215 21:05:48.260321 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 6.
I20251215 21:05:48.260377 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 6.
I20251215 21:05:48.260628 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:05:48.260854 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 6 FOLLOWER]: Leader election won for term 6
I20251215 21:05:48.261030 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 6 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:48.261171 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.262559 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 3 to 6, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 6 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:48.345779 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 6 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 6 index: 41. (index mismatch)
I20251215 21:05:48.346240 24001 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251215 21:05:48.346524 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 6 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 6 index: 41. (index mismatch)
I20251215 21:05:48.346837 24001 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251215 21:05:48.449663 23956 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:48.449798 23956 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 6 FOLLOWER]: Advancing to term 7
I20251215 21:05:48.450810 23956 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.451170 23956 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 7 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:48.451489 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 41 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:48.451496 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 41 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:48.451870 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:48.452039 23956 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251215 21:05:48.453912 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 7 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 6. Current term is 7. Ops: []
I20251215 21:05:48.454222 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 6.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251215 21:05:48.506672 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 6 LEADER]: Stepping down as leader of term 6
I20251215 21:05:48.506788 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:48.506953 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 6.41, Last appended by leader: 41, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.507130 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 6 FOLLOWER]: Advancing to term 7
I20251215 21:05:48.551234 23986 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:48.551367 23986 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 6 FOLLOWER]: Advancing to term 7
I20251215 21:05:48.552242 23986 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.552623 23986 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 7 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:48.553076 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 7 candidate_status { last_received { term: 6 index: 41 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.553076 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 7 candidate_status { last_received { term: 6 index: 41 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:48.553251 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 7: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:48.553908 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 7.
I20251215 21:05:48.554172 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:48.554344 23986 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 21:05:48.554507 23986 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 7 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:48.554664 23986 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 6.41, Last appended by leader: 41, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.556079 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 6 to 7, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 7 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:48.570573 23956 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:48.570668 23956 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 7 FOLLOWER]: Advancing to term 8
I20251215 21:05:48.571475 23956 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.571830 23956 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 8 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:48.572181 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 41 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:48.572181 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 41 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:48.572330 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 7 FOLLOWER]: Advancing to term 8
I20251215 21:05:48.573064 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 8.
I20251215 21:05:48.573280 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:48.573421 23956 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 FOLLOWER]: Leader election won for term 8
I20251215 21:05:48.573566 23956 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:48.573693 23956 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 6.41, Last appended by leader: 41, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.633563 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 7. Current term is 8. Ops: []
I20251215 21:05:48.633965 23986 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251215 21:05:48.634119 23986 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 8.42 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 41
I20251215 21:05:48.651039 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 8 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 7. Current term is 8. Ops: []
I20251215 21:05:48.651433 23986 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251215 21:05:48.653138 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 7 to 8, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 8 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:48.674351 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 8 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 41. Preceding OpId from leader: term: 8 index: 42. (index mismatch)
I20251215 21:05:48.674741 23956 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251215 21:05:48.684104 23902 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 21:05:48.684167 23902 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:48.684288 23902 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 7.42, Last appended by leader: 42, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.684427 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 7 FOLLOWER]: Advancing to term 8
I20251215 21:05:48.685288 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 8 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 42. Preceding OpId from leader: term: 8 index: 42. (term mismatch)
I20251215 21:05:48.685341 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 41
I20251215 21:05:48.685381 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 7.42
I20251215 21:05:48.685424 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:48.685755 23956 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251215 21:05:48.778239 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 8 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:48.778378 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 8 FOLLOWER]: Advancing to term 9
I20251215 21:05:48.779225 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.779605 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 9 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:48.780002 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.780005 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:48.780392 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:48.780543 24001 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251215 21:05:48.790902 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 9 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 8. Current term is 9. Ops: []
I20251215 21:05:48.791318 24011 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 8.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251215 21:05:48.865280 24010 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 LEADER]: Stepping down as leader of term 8
I20251215 21:05:48.865401 24010 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:48.865588 24010 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 8.42, Last appended by leader: 42, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.865793 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 8 FOLLOWER]: Advancing to term 9
I20251215 21:05:48.909724 24009 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:48.909857 24009 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 8 FOLLOWER]: Advancing to term 9
I20251215 21:05:48.910732 24009 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.911137 24009 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 9 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:48.911490 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.911571 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:48.911725 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 9: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:05:48.912232 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 9.
I20251215 21:05:48.912508 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:48.912684 24009 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 9 FOLLOWER]: Leader election won for term 9
I20251215 21:05:48.912766 24009 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 9 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:48.912926 24009 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 8.42, Last appended by leader: 42, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.914146 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 8 to 9, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 9 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:48.916507 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:48.916579 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 9 FOLLOWER]: Advancing to term 10
I20251215 21:05:48.917279 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.917572 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 10 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:48.917833 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 10 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:48.917845 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 10 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:48.917948 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 9 FOLLOWER]: Advancing to term 10
I20251215 21:05:48.918605 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 10.
I20251215 21:05:48.918848 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:48.918994 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 FOLLOWER]: Leader election won for term 10
I20251215 21:05:48.919073 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:48.919174 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 8.42, Last appended by leader: 42, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:48.994211 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 9 to 10, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 10 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:48.998443 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 10 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 8 index: 42. Preceding OpId from leader: term: 10 index: 43. (index mismatch)
I20251215 21:05:48.998792 24001 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251215 21:05:49.012307 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 10 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 9. Current term is 10. Ops: []
I20251215 21:05:49.012646 24009 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 43, Time since last communication: 0.000s
I20251215 21:05:49.017192 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 9. Current term is 10. Ops: []
I20251215 21:05:49.017529 23986 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 43, Time since last communication: 0.000s
I20251215 21:05:49.017637 23986 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 10.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 43
I20251215 21:05:49.018229 23902 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 9 LEADER]: Stepping down as leader of term 9
I20251215 21:05:49.018286 23902 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:49.018407 23902 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 9.43, Last appended by leader: 43, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.018550 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 9 FOLLOWER]: Advancing to term 10
I20251215 21:05:49.019433 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 10 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 9 index: 43. Preceding OpId from leader: term: 10 index: 43. (term mismatch)
I20251215 21:05:49.019480 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 42
I20251215 21:05:49.019501 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 9.43
I20251215 21:05:49.019524 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:49.019879 24001 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251215 21:05:49.101387 24010 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:49.101514 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 10 FOLLOWER]: Advancing to term 11
I20251215 21:05:49.102368 24010 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.102746 24010 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 11 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.103128 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.103132 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:49.103513 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:49.103672 24010 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251215 21:05:49.120990 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 11 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 10. Current term is 11. Ops: []
I20251215 21:05:49.121372 24008 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 10.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251215 21:05:49.205150 24010 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:49.205282 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 11 FOLLOWER]: Advancing to term 12
I20251215 21:05:49.206133 24010 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.206540 24010 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 12 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.206938 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.206939 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:49.207314 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:49.207472 24010 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251215 21:05:49.210197 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 10. Current term is 12. Ops: []
I20251215 21:05:49.210522 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 10.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251215 21:05:49.252017 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 LEADER]: Stepping down as leader of term 10
I20251215 21:05:49.252143 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:49.252336 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 10.43, Last appended by leader: 43, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.252544 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 10 FOLLOWER]: Advancing to term 11
I20251215 21:05:49.261607 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.262940 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.265129 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.269307 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.277495 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.293708 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.313870 24009 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 10 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:49.314009 24009 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 10 FOLLOWER]: Advancing to term 11
I20251215 21:05:49.314913 24009 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.315312 24009 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 11 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.315739 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 11 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:49.315743 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 11 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.315867 23826 raft_consensus.cc:2368] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for earlier term 11. Current term is 12.
I20251215 21:05:49.316160 23868 leader_election.cc:400] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 11 election: Vote denied by peer ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057) with higher term. Message: Invalid argument: T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for earlier term 11. Current term is 12.
I20251215 21:05:49.316200 23868 leader_election.cc:403] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20251215 21:05:49.316308 24009 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 11 FOLLOWER]: Advancing to term 12
I20251215 21:05:49.316663 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 11.
I20251215 21:05:49.317003 24009 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057) with higher term. Message: Invalid argument: T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for earlier term 11. Current term is 12.
I20251215 21:05:49.325994 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.370129 24010 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:49.370266 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 12 FOLLOWER]: Advancing to term 13
I20251215 21:05:49.371170 24010 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.371625 24010 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 13 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.372079 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.372079 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 43 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:49.372224 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 11 FOLLOWER]: Advancing to term 13
I20251215 21:05:49.372224 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 12 FOLLOWER]: Advancing to term 13
I20251215 21:05:49.373193 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 13.
I20251215 21:05:49.373193 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 13.
I20251215 21:05:49.373521 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:05:49.373684 24010 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 13 FOLLOWER]: Leader election won for term 13
I20251215 21:05:49.373847 24010 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 13 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:49.373991 24010 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 10.43, Last appended by leader: 43, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.375325 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 10 to 13, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 13 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:49.390277 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.459508 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 13 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 43. Preceding OpId from leader: term: 13 index: 44. (index mismatch)
I20251215 21:05:49.459883 24010 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251215 21:05:49.462179 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 13 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 43. Preceding OpId from leader: term: 13 index: 44. (index mismatch)
I20251215 21:05:49.462529 24010 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251215 21:05:49.518576 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.563588 24009 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:49.563704 24009 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 13 FOLLOWER]: Advancing to term 14
I20251215 21:05:49.564519 24009 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.564916 24009 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 14 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.565223 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 14 candidate_status { last_received { term: 13 index: 44 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:49.565286 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 14 candidate_status { last_received { term: 13 index: 44 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.565636 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 14 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 13. Current term is 14. Ops: []
I20251215 21:05:49.565754 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:49.565934 24009 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20251215 21:05:49.566107 24015 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 13.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251215 21:05:49.566284 24010 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 13 LEADER]: Stepping down as leader of term 13
I20251215 21:05:49.566340 24010 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:49.566475 24010 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 13.44, Last appended by leader: 44, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.566612 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 13 FOLLOWER]: Advancing to term 14
I20251215 21:05:49.658232 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 13 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:49.658372 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 13 FOLLOWER]: Advancing to term 14
I20251215 21:05:49.659235 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.659627 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 14 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:49.660024 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 14 candidate_status { last_received { term: 13 index: 44 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:49.660022 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 14 candidate_status { last_received { term: 13 index: 44 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:49.660202 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 14: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:49.660912 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 14.
I20251215 21:05:49.661152 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:49.661298 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 14 FOLLOWER]: Leader election won for term 14
I20251215 21:05:49.661384 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 14 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:49.661520 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 13.44, Last appended by leader: 44, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.662743 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 13 to 14, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 14 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:49.667613 24009 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:49.667687 24009 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 14 FOLLOWER]: Advancing to term 15
I20251215 21:05:49.668452 24009 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.668830 24009 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 15 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.669171 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 15 candidate_status { last_received { term: 13 index: 44 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.669190 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 15 candidate_status { last_received { term: 13 index: 44 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:49.669297 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 14 FOLLOWER]: Advancing to term 15
I20251215 21:05:49.669986 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 15.
I20251215 21:05:49.670229 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:49.670382 24009 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 15 FOLLOWER]: Leader election won for term 15
I20251215 21:05:49.670540 24009 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 15 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:49.670673 24009 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 13.44, Last appended by leader: 44, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.671895 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 14 to 15, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 15 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:49.737957 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 15 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 14. Current term is 15. Ops: []
I20251215 21:05:49.738404 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251215 21:05:49.738587 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 14 LEADER]: Stepping down as leader of term 14
I20251215 21:05:49.738632 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:49.738760 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 14.45, Last appended by leader: 45, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.738899 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 14 FOLLOWER]: Advancing to term 15
I20251215 21:05:49.774873 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:49.790071 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 15 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 13 index: 44. Preceding OpId from leader: term: 15 index: 45. (index mismatch)
I20251215 21:05:49.790436 24009 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251215 21:05:49.792392 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 15 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 14 index: 45. Preceding OpId from leader: term: 15 index: 45. (term mismatch)
I20251215 21:05:49.792454 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 44
I20251215 21:05:49.792479 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 14.45
I20251215 21:05:49.792500 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:49.792958 24016 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251215 21:05:49.978403 24010 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:49.978540 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 15 FOLLOWER]: Advancing to term 16
I20251215 21:05:49.979431 24010 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:49.979847 24010 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 16 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:49.980268 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 45 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:49.980268 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 45 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:49.980652 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:49.980847 24010 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20251215 21:05:50.000211 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 16 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 15. Current term is 16. Ops: []
I20251215 21:05:50.000603 24009 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 15.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251215 21:05:50.000830 24016 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 15 LEADER]: Stepping down as leader of term 15
I20251215 21:05:50.000890 24016 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:50.001015 24016 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 15.45, Last appended by leader: 45, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.001160 24016 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 15 FOLLOWER]: Advancing to term 16
I20251215 21:05:50.007359 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:50.007457 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 15 FOLLOWER]: Advancing to term 16
I20251215 21:05:50.008198 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.008553 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 16 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:50.008890 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 16 candidate_status { last_received { term: 15 index: 45 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:50.008888 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 16 candidate_status { last_received { term: 15 index: 45 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:50.009045 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 16: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:50.009723 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 16.
I20251215 21:05:50.009958 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:50.010123 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 16 FOLLOWER]: Leader election won for term 16
I20251215 21:05:50.010203 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 16 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:50.010330 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 15.45, Last appended by leader: 45, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.011567 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 15 to 16, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 16 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:50.100853 24010 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:50.100979 24010 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 16 FOLLOWER]: Advancing to term 17
I20251215 21:05:50.101817 24010 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.102233 24010 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 17 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:50.102638 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 45 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:50.102638 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 45 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:50.102762 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 16 FOLLOWER]: Advancing to term 17
I20251215 21:05:50.103560 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 17.
I20251215 21:05:50.103801 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:50.103937 24010 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 17 FOLLOWER]: Leader election won for term 17
I20251215 21:05:50.104104 24010 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 17 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:50.104249 24010 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 15.45, Last appended by leader: 45, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.104547 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 17 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 16. Current term is 17. Ops: []
I20251215 21:05:50.104882 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251215 21:05:50.105019 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 16 LEADER]: Stepping down as leader of term 16
I20251215 21:05:50.105063 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:50.105180 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 16.46, Last appended by leader: 46, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.105304 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 16 FOLLOWER]: Advancing to term 17
I20251215 21:05:50.105830 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 16 to 17, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 17 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:50.199831 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 17 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 45. Preceding OpId from leader: term: 17 index: 46. (index mismatch)
I20251215 21:05:50.200232 24010 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251215 21:05:50.202636 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 17 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 46. Preceding OpId from leader: term: 17 index: 46. (term mismatch)
I20251215 21:05:50.202711 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 45
I20251215 21:05:50.202734 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 16.46
I20251215 21:05:50.202757 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:50.203065 24019 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251215 21:05:50.287195 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:50.490446 24016 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:50.490581 24016 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 17 FOLLOWER]: Advancing to term 18
I20251215 21:05:50.491780 24016 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.492368 24016 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 18 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:50.492749 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 18 candidate_status { last_received { term: 17 index: 46 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:50.492746 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 18 candidate_status { last_received { term: 17 index: 46 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:50.493207 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:50.493377 24016 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20251215 21:05:50.499176 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 18 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 17. Current term is 18. Ops: []
I20251215 21:05:50.499567 24018 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 17.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251215 21:05:50.499759 24019 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 17 LEADER]: Stepping down as leader of term 17
I20251215 21:05:50.499815 24019 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:50.499938 24019 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 17.46, Last appended by leader: 46, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.500088 24019 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 17 FOLLOWER]: Advancing to term 18
I20251215 21:05:50.565526 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 17 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:50.565653 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 17 FOLLOWER]: Advancing to term 18
I20251215 21:05:50.566498 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.566884 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 18 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:50.567303 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 18 candidate_status { last_received { term: 17 index: 46 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:50.567312 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 18 candidate_status { last_received { term: 17 index: 46 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:50.567440 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 18: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:50.568153 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 18.
I20251215 21:05:50.568408 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:50.568595 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 18 FOLLOWER]: Leader election won for term 18
I20251215 21:05:50.568688 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 18 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:50.568857 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 17.46, Last appended by leader: 46, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.570158 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 17 to 18, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 18 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:50.599903 24016 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:50.600049 24016 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 18 FOLLOWER]: Advancing to term 19
I20251215 21:05:50.600991 24016 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.601413 24016 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 19 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:50.601826 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 19 candidate_status { last_received { term: 17 index: 46 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:50.601826 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 19 candidate_status { last_received { term: 17 index: 46 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:50.602027 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 18 FOLLOWER]: Advancing to term 19
I20251215 21:05:50.602814 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 19.
I20251215 21:05:50.603070 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:50.603243 24016 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 19 FOLLOWER]: Leader election won for term 19
I20251215 21:05:50.603396 24016 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 19 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:50.603541 24016 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 17.46, Last appended by leader: 46, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.604910 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 18 to 19, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 19 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:50.677459 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 19 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 18. Current term is 19. Ops: []
I20251215 21:05:50.677886 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251215 21:05:50.678063 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 18 LEADER]: Stepping down as leader of term 18
I20251215 21:05:50.678113 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:50.678257 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 18.47, Last appended by leader: 47, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.678421 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 18 FOLLOWER]: Advancing to term 19
I20251215 21:05:50.685843 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 19 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 17 index: 46. Preceding OpId from leader: term: 19 index: 47. (index mismatch)
I20251215 21:05:50.686218 24016 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251215 21:05:50.688110 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 19 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 18 index: 47. Preceding OpId from leader: term: 19 index: 47. (term mismatch)
I20251215 21:05:50.688172 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 46
I20251215 21:05:50.688194 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 18.47
I20251215 21:05:50.688216 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:50.688959 24021 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251215 21:05:50.789234 24019 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:50.789381 24019 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 19 FOLLOWER]: Advancing to term 20
I20251215 21:05:50.790387 24019 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.791213 24019 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 20 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:50.791313 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 47 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:50.791438 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 47 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:50.791826 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:50.792022 24019 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20251215 21:05:50.810401 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 20 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 19. Current term is 20. Ops: []
I20251215 21:05:50.810849 24021 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 19.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251215 21:05:50.811085 24020 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 19 LEADER]: Stepping down as leader of term 19
I20251215 21:05:50.811144 24020 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:50.811282 24020 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 19.47, Last appended by leader: 47, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.811437 24020 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 19 FOLLOWER]: Advancing to term 20
I20251215 21:05:50.903849 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:50.903985 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 19 FOLLOWER]: Advancing to term 20
I20251215 21:05:50.904909 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.905303 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 20 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:50.905691 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 20 candidate_status { last_received { term: 19 index: 47 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:50.905689 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 20 candidate_status { last_received { term: 19 index: 47 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:50.905844 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 20: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:50.906579 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 20.
I20251215 21:05:50.906826 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:50.906978 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 20 FOLLOWER]: Leader election won for term 20
I20251215 21:05:50.907068 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 20 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:50.907182 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 19.47, Last appended by leader: 47, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.908440 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 19 to 20, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 20 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:50.935545 24019 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:50.935698 24019 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 20 FOLLOWER]: Advancing to term 21
I20251215 21:05:50.936652 24019 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.937163 24019 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 21 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:50.937603 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 47 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:50.937601 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 47 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:50.937765 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 20 FOLLOWER]: Advancing to term 21
I20251215 21:05:50.938607 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 21.
I20251215 21:05:50.938892 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:50.939145 24019 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 FOLLOWER]: Leader election won for term 21
I20251215 21:05:50.939235 24019 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:50.939405 24019 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 19.47, Last appended by leader: 47, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:50.940755 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 20 to 21, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 21 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:51.007669 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 21 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 20. Current term is 21. Ops: []
I20251215 21:05:51.008123 24008 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251215 21:05:51.008359 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 20 LEADER]: Stepping down as leader of term 20
I20251215 21:05:51.008424 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:51.008517 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 20. Current term is 21. Ops: []
I20251215 21:05:51.008555 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 20.48, Last appended by leader: 48, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.008701 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 20 FOLLOWER]: Advancing to term 21
I20251215 21:05:51.031944 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 21 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 48. Preceding OpId from leader: term: 21 index: 48. (term mismatch)
I20251215 21:05:51.032043 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 47
I20251215 21:05:51.032063 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 20.48
I20251215 21:05:51.032083 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:51.032506 24019 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251215 21:05:51.034586 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 21 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 47. Preceding OpId from leader: term: 21 index: 48. (index mismatch)
I20251215 21:05:51.035043 24018 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251215 21:05:51.136219 24020 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:51.136368 24020 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 21 FOLLOWER]: Advancing to term 22
I20251215 21:05:51.137404 24020 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.137845 24020 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 22 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:51.138271 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 22 candidate_status { last_received { term: 21 index: 48 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:51.138268 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 22 candidate_status { last_received { term: 21 index: 48 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:51.138659 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:51.138816 24020 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20251215 21:05:51.155634 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 22 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 21. Current term is 22. Ops: []
I20251215 21:05:51.156092 24022 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 21.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251215 21:05:51.156279 24022 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 LEADER]: Stepping down as leader of term 21
I20251215 21:05:51.156323 24022 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:51.156456 24022 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 21.48, Last appended by leader: 48, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.156617 24022 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 21 FOLLOWER]: Advancing to term 22
I20251215 21:05:51.234459 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 21 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:51.234592 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 21 FOLLOWER]: Advancing to term 22
I20251215 21:05:51.235488 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.235903 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 22 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:51.236449 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 22 candidate_status { last_received { term: 21 index: 48 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:51.236449 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 22 candidate_status { last_received { term: 21 index: 48 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:51.236635 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 22: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:51.237494 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 22.
I20251215 21:05:51.237815 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:51.238013 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 22 FOLLOWER]: Leader election won for term 22
I20251215 21:05:51.238102 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 22 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:51.238250 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 21.48, Last appended by leader: 48, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.239593 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 21 to 22, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 22 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:51.253253 24020 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:51.253373 24020 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 22 FOLLOWER]: Advancing to term 23
I20251215 21:05:51.254201 24020 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.254592 24020 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 23 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:51.254942 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 23 candidate_status { last_received { term: 21 index: 48 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:51.254978 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 23 candidate_status { last_received { term: 21 index: 48 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:51.255059 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 22 FOLLOWER]: Advancing to term 23
I20251215 21:05:51.255744 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 23.
I20251215 21:05:51.255960 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:51.256103 24020 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 FOLLOWER]: Leader election won for term 23
I20251215 21:05:51.256191 24020 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:51.256336 24020 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 21.48, Last appended by leader: 48, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.257591 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 22 to 23, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 23 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:51.287480 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:51.340986 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 22. Current term is 23. Ops: []
I20251215 21:05:51.341464 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251215 21:05:51.341616 24001 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 23.49 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 48
I20251215 21:05:51.341678 24008 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 22 LEADER]: Stepping down as leader of term 22
I20251215 21:05:51.341742 24008 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 22 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:51.342003 24008 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 22.49, Last appended by leader: 49, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.342193 24008 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 22 FOLLOWER]: Advancing to term 23
I20251215 21:05:51.354815 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 23 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 21 index: 48. Preceding OpId from leader: term: 23 index: 49. (index mismatch)
I20251215 21:05:51.355283 24020 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251215 21:05:51.357415 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 23 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 22 index: 49. Preceding OpId from leader: term: 23 index: 49. (term mismatch)
I20251215 21:05:51.357486 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 48
I20251215 21:05:51.357510 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 22.49
I20251215 21:05:51.357540 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:51.357998 24021 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251215 21:05:51.556155 24022 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:51.556288 24022 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 23 FOLLOWER]: Advancing to term 24
I20251215 21:05:51.557215 24022 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.557595 24022 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 24 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:51.557996 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 49 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:51.558012 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 49 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:51.558360 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:51.558514 24022 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20251215 21:05:51.579257 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 24 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 23. Current term is 24. Ops: []
I20251215 21:05:51.579622 24023 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 23.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251215 21:05:51.579818 24021 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 LEADER]: Stepping down as leader of term 23
I20251215 21:05:51.579887 24021 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:51.580025 24021 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 23.49, Last appended by leader: 49, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.580175 24021 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 23 FOLLOWER]: Advancing to term 24
I20251215 21:05:51.646952 24008 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:51.647105 24008 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 23 FOLLOWER]: Advancing to term 24
I20251215 21:05:51.648000 24008 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.648429 24008 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 24 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:51.648860 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 24 candidate_status { last_received { term: 23 index: 49 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:51.648888 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 24 candidate_status { last_received { term: 23 index: 49 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:51.649010 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 24: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:51.649737 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 24.
I20251215 21:05:51.650043 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:51.650209 24008 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 24 FOLLOWER]: Leader election won for term 24
I20251215 21:05:51.650292 24008 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 24 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:51.650434 24008 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 23.49, Last appended by leader: 49, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.651705 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 23 to 24, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 24 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:51.688540 24022 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:51.688661 24022 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 24 FOLLOWER]: Advancing to term 25
I20251215 21:05:51.689625 24022 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.690008 24022 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 25 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:51.690412 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 49 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:51.690410 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 49 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:51.690552 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 24 FOLLOWER]: Advancing to term 25
I20251215 21:05:51.691251 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 25.
I20251215 21:05:51.691498 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:51.691669 24022 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 FOLLOWER]: Leader election won for term 25
I20251215 21:05:51.691836 24022 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:51.691984 24022 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 23.49, Last appended by leader: 49, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.693315 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 24 to 25, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 25 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:51.764030 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 24. Current term is 25. Ops: []
I20251215 21:05:51.764453 24008 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251215 21:05:51.764596 24008 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 25.50 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 49
I20251215 21:05:51.764653 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 24 LEADER]: Stepping down as leader of term 24
I20251215 21:05:51.764703 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:51.764855 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 24.50, Last appended by leader: 50, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:51.764983 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 24 FOLLOWER]: Advancing to term 25
I20251215 21:05:51.769416 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 25 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 49. Preceding OpId from leader: term: 25 index: 50. (index mismatch)
I20251215 21:05:51.769755 24022 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251215 21:05:51.771744 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 25 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 50. Preceding OpId from leader: term: 25 index: 50. (term mismatch)
I20251215 21:05:51.771803 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 49
I20251215 21:05:51.771827 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 24.50
I20251215 21:05:51.771849 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:51.772167 24024 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251215 21:05:52.263703 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 25 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:52.263835 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 25 FOLLOWER]: Advancing to term 26
I20251215 21:05:52.264699 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.265168 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 26 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:52.265588 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 26 candidate_status { last_received { term: 25 index: 50 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:52.265585 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 26 candidate_status { last_received { term: 25 index: 50 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:52.265941 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:52.266090 24001 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20251215 21:05:52.277583 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 26 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 25. Current term is 26. Ops: []
I20251215 21:05:52.277907 24024 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 25.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251215 21:05:52.278084 24025 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 LEADER]: Stepping down as leader of term 25
I20251215 21:05:52.278136 24025 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:52.278282 24025 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 25.50, Last appended by leader: 50, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.278431 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 25 FOLLOWER]: Advancing to term 26
I20251215 21:05:52.287760 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:52.298087 24026 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:52.298162 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 25 FOLLOWER]: Advancing to term 26
I20251215 21:05:52.299045 24026 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.299394 24026 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 26 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:52.299739 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 26 candidate_status { last_received { term: 25 index: 50 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:52.299754 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 26 candidate_status { last_received { term: 25 index: 50 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:52.299886 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 26: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:05:52.300565 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 26.
I20251215 21:05:52.300812 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:52.300976 24026 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 26 FOLLOWER]: Leader election won for term 26
I20251215 21:05:52.301137 24026 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 26 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:52.301265 24026 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 25.50, Last appended by leader: 50, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.302662 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 25 to 26, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 26 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:52.393438 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:52.393568 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 26 FOLLOWER]: Advancing to term 27
I20251215 21:05:52.394421 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.394793 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 27 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:52.395202 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 27 candidate_status { last_received { term: 25 index: 50 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:52.395211 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 27 candidate_status { last_received { term: 25 index: 50 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:52.395321 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 26 FOLLOWER]: Advancing to term 27
I20251215 21:05:52.396167 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 27.
I20251215 21:05:52.396428 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:52.396443 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 26. Current term is 27. Ops: []
I20251215 21:05:52.396644 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 FOLLOWER]: Leader election won for term 27
I20251215 21:05:52.396817 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:52.396785 24026 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251215 21:05:52.396957 24026 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 26 LEADER]: Stepping down as leader of term 26
I20251215 21:05:52.396986 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 25.50, Last appended by leader: 50, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.397044 24026 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:52.397186 24026 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 26.51, Last appended by leader: 51, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.397328 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 26 FOLLOWER]: Advancing to term 27
I20251215 21:05:52.398473 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 26 to 27, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 27 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:52.491477 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 27 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 26 index: 51. Preceding OpId from leader: term: 27 index: 51. (term mismatch)
I20251215 21:05:52.491565 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 50
I20251215 21:05:52.491582 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 26.51
I20251215 21:05:52.491600 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:52.491976 24001 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251215 21:05:52.493916 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 27 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 25 index: 50. Preceding OpId from leader: term: 27 index: 51. (index mismatch)
I20251215 21:05:52.494410 24031 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251215 21:05:52.683604 24025 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:52.683707 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 27 FOLLOWER]: Advancing to term 28
I20251215 21:05:52.684499 24025 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.684847 24025 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 28 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:52.685137 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 51 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:52.685137 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 51 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:52.685482 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:52.685624 24025 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20251215 21:05:52.697255 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 28 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 27. Current term is 28. Ops: []
I20251215 21:05:52.697618 24032 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 27.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251215 21:05:52.697817 24001 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 LEADER]: Stepping down as leader of term 27
I20251215 21:05:52.697882 24001 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:52.698011 24001 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 27.51, Last appended by leader: 51, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.698154 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 27 FOLLOWER]: Advancing to term 28
I20251215 21:05:52.782873 24026 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 27 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:52.783003 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 27 FOLLOWER]: Advancing to term 28
I20251215 21:05:52.783864 24026 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.784265 24026 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 28 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:52.784703 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 28 candidate_status { last_received { term: 27 index: 51 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:52.784703 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 28 candidate_status { last_received { term: 27 index: 51 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:52.784909 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 28: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:52.785670 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 28.
I20251215 21:05:52.785926 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:52.786079 24026 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 28 FOLLOWER]: Leader election won for term 28
I20251215 21:05:52.786166 24026 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 28 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:52.786352 24026 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 27.51, Last appended by leader: 51, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.787770 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 27 to 28, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 28 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:52.828086 24025 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:52.828217 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 28 FOLLOWER]: Advancing to term 29
I20251215 21:05:52.829103 24025 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.829504 24025 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 29 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:52.829900 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 51 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:52.829921 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 51 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:52.830015 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 28 FOLLOWER]: Advancing to term 29
I20251215 21:05:52.830741 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 29.
I20251215 21:05:52.830979 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:52.831130 24025 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 FOLLOWER]: Leader election won for term 29
I20251215 21:05:52.831287 24025 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:52.831413 24025 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 27.51, Last appended by leader: 51, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.832679 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 28 to 29, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 29 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:52.873764 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 28. Current term is 29. Ops: []
I20251215 21:05:52.874143 24026 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251215 21:05:52.874295 24026 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 29.52 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I20251215 21:05:52.874353 24027 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 28 LEADER]: Stepping down as leader of term 28
I20251215 21:05:52.874405 24027 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:52.874534 24027 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 28.52, Last appended by leader: 52, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:52.874671 24027 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 28 FOLLOWER]: Advancing to term 29
I20251215 21:05:52.940515 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 29 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 51. Preceding OpId from leader: term: 29 index: 52. (index mismatch)
I20251215 21:05:52.940919 24025 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251215 21:05:52.943130 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 29 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 52. Preceding OpId from leader: term: 29 index: 52. (term mismatch)
I20251215 21:05:52.943199 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 51
I20251215 21:05:52.943224 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 28.52
I20251215 21:05:52.943248 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:52.943542 24034 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251215 21:05:53.155894 24026 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 29 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:53.156024 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 29 FOLLOWER]: Advancing to term 30
I20251215 21:05:53.156931 24026 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.157338 24026 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 30 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:53.157717 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 30 candidate_status { last_received { term: 29 index: 52 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:53.157725 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 30 candidate_status { last_received { term: 29 index: 52 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:53.158080 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:53.158232 24026 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20251215 21:05:53.176875 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 30 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 29. Current term is 30. Ops: []
I20251215 21:05:53.177244 24034 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 29.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251215 21:05:53.177429 24025 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 LEADER]: Stepping down as leader of term 29
I20251215 21:05:53.177484 24025 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:53.177621 24025 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 29.52, Last appended by leader: 52, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.177770 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 29 FOLLOWER]: Advancing to term 30
I20251215 21:05:53.219156 24001 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 29 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:53.219296 24001 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 29 FOLLOWER]: Advancing to term 30
I20251215 21:05:53.220170 24001 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.220578 24001 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 30 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:53.221029 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 30 candidate_status { last_received { term: 29 index: 52 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:53.221035 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 30 candidate_status { last_received { term: 29 index: 52 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:53.221170 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 30: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:53.221932 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 30.
I20251215 21:05:53.222210 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:53.222389 24001 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 30 FOLLOWER]: Leader election won for term 30
I20251215 21:05:53.222575 24001 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 30 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:53.222716 24001 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 29.52, Last appended by leader: 52, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.224012 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 29 to 30, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 30 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:53.258637 24026 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:53.258759 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 30 FOLLOWER]: Advancing to term 31
I20251215 21:05:53.259631 24026 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.260042 24026 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 31 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:53.260413 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 31 candidate_status { last_received { term: 29 index: 52 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:53.260437 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 31 candidate_status { last_received { term: 29 index: 52 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:53.260510 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 30 FOLLOWER]: Advancing to term 31
I20251215 21:05:53.261246 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 31.
I20251215 21:05:53.261478 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:53.261622 24026 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 FOLLOWER]: Leader election won for term 31
I20251215 21:05:53.261729 24026 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:53.261878 24026 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 29.52, Last appended by leader: 52, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.263203 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 30 to 31, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 31 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:53.288009 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:53.306146 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 30. Current term is 31. Ops: []
I20251215 21:05:53.306565 24001 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251215 21:05:53.306704 24001 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 31.53 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 52
I20251215 21:05:53.306766 24035 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 30 LEADER]: Stepping down as leader of term 30
I20251215 21:05:53.306833 24035 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:53.306979 24035 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 30.53, Last appended by leader: 53, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.307128 24035 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 30 FOLLOWER]: Advancing to term 31
I20251215 21:05:53.370424 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 31 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 30 index: 53. Preceding OpId from leader: term: 31 index: 53. (term mismatch)
I20251215 21:05:53.370513 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 52
I20251215 21:05:53.370529 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 30.53
I20251215 21:05:53.370546 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:53.370909 24026 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251215 21:05:53.372762 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 31 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 29 index: 52. Preceding OpId from leader: term: 31 index: 53. (index mismatch)
I20251215 21:05:53.373121 24026 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251215 21:05:53.553671 24025 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:53.553825 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 31 FOLLOWER]: Advancing to term 32
I20251215 21:05:53.554759 24025 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.555189 24025 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 32 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:53.555619 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 53 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:53.555619 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 53 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:53.556015 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:53.556162 24025 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20251215 21:05:53.571031 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 31. Current term is 32. Ops: []
I20251215 21:05:53.571486 24036 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 31.53, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251215 21:05:53.571682 24026 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 LEADER]: Stepping down as leader of term 31
I20251215 21:05:53.571739 24026 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:53.571861 24026 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 31.53, Last appended by leader: 53, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.572016 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 31 FOLLOWER]: Advancing to term 32
I20251215 21:05:53.589579 24035 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:53.589707 24035 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 31 FOLLOWER]: Advancing to term 32
I20251215 21:05:53.590571 24035 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.590948 24035 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 32 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:53.591320 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 32 candidate_status { last_received { term: 31 index: 53 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:53.591346 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 32 candidate_status { last_received { term: 31 index: 53 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:53.591486 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 32: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:53.592164 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 32.
I20251215 21:05:53.592425 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:53.592578 24035 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 32 FOLLOWER]: Leader election won for term 32
I20251215 21:05:53.592671 24035 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 32 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:53.592830 24035 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 31.53, Last appended by leader: 53, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.594136 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 31 to 32, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 32 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:53.683305 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 32 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 31 index: 53. Preceding OpId from leader: term: 32 index: 54. (index mismatch)
I20251215 21:05:53.683681 24035 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251215 21:05:53.685883 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 31 index: 53. Preceding OpId from leader: term: 32 index: 54. (index mismatch)
I20251215 21:05:53.686250 24037 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251215 21:05:53.787923 24025 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:53.788049 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 32 FOLLOWER]: Advancing to term 33
I20251215 21:05:53.788926 24025 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.789297 24025 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 33 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:53.789575 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 54 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:53.789603 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 54 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:53.789911 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:53.790061 24025 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20251215 21:05:53.808864 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 33 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 32. Current term is 33. Ops: []
I20251215 21:05:53.809285 24038 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 32.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251215 21:05:53.809484 24038 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 32 LEADER]: Stepping down as leader of term 32
I20251215 21:05:53.809540 24038 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:53.809675 24038 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 32.54, Last appended by leader: 54, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.809851 24038 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 32 FOLLOWER]: Advancing to term 33
I20251215 21:05:53.888720 24026 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 32 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:53.888900 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 32 FOLLOWER]: Advancing to term 33
I20251215 21:05:53.889808 24026 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.890199 24026 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 33 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:53.890599 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 33 candidate_status { last_received { term: 32 index: 54 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:53.890605 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 33 candidate_status { last_received { term: 32 index: 54 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:53.890749 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 33: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:53.891454 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 33.
I20251215 21:05:53.891706 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:53.891861 24026 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 33 FOLLOWER]: Leader election won for term 33
I20251215 21:05:53.891948 24026 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 33 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:53.892091 24026 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 32.54, Last appended by leader: 54, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.893404 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 32 to 33, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 33 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:53.910254 24025 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:53.910389 24025 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 33 FOLLOWER]: Advancing to term 34
I20251215 21:05:53.911227 24025 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.911618 24025 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 34 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:53.911999 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 54 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:53.911996 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 54 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:53.912132 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 33 FOLLOWER]: Advancing to term 34
I20251215 21:05:53.912889 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 34.
I20251215 21:05:53.913151 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:53.913352 24025 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 34 FOLLOWER]: Leader election won for term 34
I20251215 21:05:53.913537 24025 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 34 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:53.913672 24025 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 32.54, Last appended by leader: 54, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:53.914929 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 33 to 34, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 34 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:54.003197 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 34 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 54. Preceding OpId from leader: term: 34 index: 55. (index mismatch)
I20251215 21:05:54.003582 24025 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251215 21:05:54.005602 23902 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 33 LEADER]: Stepping down as leader of term 33
I20251215 21:05:54.005667 23902 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:54.005797 23902 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 33.55, Last appended by leader: 55, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.005941 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 33 FOLLOWER]: Advancing to term 34
I20251215 21:05:54.006995 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 34 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 33 index: 55. Preceding OpId from leader: term: 34 index: 55. (term mismatch)
I20251215 21:05:54.007040 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 54
I20251215 21:05:54.007055 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 33.55
I20251215 21:05:54.007073 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:54.007391 24040 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251215 21:05:54.288295 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:54.293848 24038 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 34 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:54.293960 24038 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 34 FOLLOWER]: Advancing to term 35
I20251215 21:05:54.294864 24038 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.295258 24038 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 35 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:54.295665 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 35 candidate_status { last_received { term: 34 index: 55 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:54.295667 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 35 candidate_status { last_received { term: 34 index: 55 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:54.296051 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:54.296206 24038 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20251215 21:05:54.307488 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 35 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 34. Current term is 35. Ops: []
I20251215 21:05:54.307886 24025 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 34.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251215 21:05:54.308079 24040 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 34 LEADER]: Stepping down as leader of term 34
I20251215 21:05:54.308135 24040 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 34 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:54.308256 24040 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 34.55, Last appended by leader: 55, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.308391 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 34 FOLLOWER]: Advancing to term 35
I20251215 21:05:54.328440 24026 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 34 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:54.328552 24026 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 34 FOLLOWER]: Advancing to term 35
I20251215 21:05:54.329488 24026 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.329869 24026 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 35 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:54.330227 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 35 candidate_status { last_received { term: 34 index: 55 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:54.330255 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 35 candidate_status { last_received { term: 34 index: 55 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:54.330358 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 35: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:05:54.331014 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 35.
I20251215 21:05:54.331265 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:54.331429 24026 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 35 FOLLOWER]: Leader election won for term 35
I20251215 21:05:54.331506 24026 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 35 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:54.331631 24026 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 34.55, Last appended by leader: 55, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.332935 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 34 to 35, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 35 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:54.428956 24038 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:54.429100 24038 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 35 FOLLOWER]: Advancing to term 36
I20251215 21:05:54.430011 24038 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.430394 24038 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 36 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:54.430771 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 36 candidate_status { last_received { term: 34 index: 55 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:54.430774 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 36 candidate_status { last_received { term: 34 index: 55 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:54.430872 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 35 FOLLOWER]: Advancing to term 36
I20251215 21:05:54.431609 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 36.
I20251215 21:05:54.431866 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:54.432011 24038 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 FOLLOWER]: Leader election won for term 36
I20251215 21:05:54.432211 24038 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:54.432372 24038 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 34.55, Last appended by leader: 55, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.432997 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 35. Current term is 36. Ops: []
I20251215 21:05:54.433449 24026 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251215 21:05:54.433553 24026 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 36.56 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 55
I20251215 21:05:54.433602 24036 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 35 LEADER]: Stepping down as leader of term 35
I20251215 21:05:54.433670 24036 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:54.433809 24036 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 35.56, Last appended by leader: 56, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.433825 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 35 to 36, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 36 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:54.433964 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 35 FOLLOWER]: Advancing to term 36
I20251215 21:05:54.512953 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 36 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 34 index: 55. Preceding OpId from leader: term: 36 index: 56. (index mismatch)
I20251215 21:05:54.513337 24038 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251215 21:05:54.515630 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 36 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 35 index: 56. Preceding OpId from leader: term: 36 index: 56. (term mismatch)
I20251215 21:05:54.515698 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 55
I20251215 21:05:54.515722 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 35.56
I20251215 21:05:54.515796 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:54.516283 24041 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251215 21:05:54.736410 24036 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 36 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:54.736534 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 36 FOLLOWER]: Advancing to term 37
I20251215 21:05:54.737473 24036 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.737869 24036 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 37 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:54.738279 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 37 candidate_status { last_received { term: 36 index: 56 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:54.738288 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 37 candidate_status { last_received { term: 36 index: 56 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:54.738652 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:54.738804 24036 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20251215 21:05:54.745311 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 36. Current term is 37. Ops: []
I20251215 21:05:54.745633 24038 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 36.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20251215 21:05:54.745796 24041 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 LEADER]: Stepping down as leader of term 36
I20251215 21:05:54.745850 24041 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:54.745965 24041 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 36.56, Last appended by leader: 56, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.746101 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 36 FOLLOWER]: Advancing to term 37
I20251215 21:05:54.778635 24040 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 36 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:54.778779 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 36 FOLLOWER]: Advancing to term 37
I20251215 21:05:54.779662 24040 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.780071 24040 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 37 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:54.780467 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 37 candidate_status { last_received { term: 36 index: 56 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:54.780475 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 37 candidate_status { last_received { term: 36 index: 56 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:54.780604 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 37: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:54.781304 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 37.
I20251215 21:05:54.781541 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:54.781690 24040 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 37 FOLLOWER]: Leader election won for term 37
I20251215 21:05:54.781770 24040 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 37 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:54.781921 24040 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 36.56, Last appended by leader: 56, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.783288 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 36 to 37, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 37 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:54.866669 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 37 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 36 index: 56. Preceding OpId from leader: term: 37 index: 57. (index mismatch)
I20251215 21:05:54.867136 24040 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20251215 21:05:54.869555 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 36 index: 56. Preceding OpId from leader: term: 37 index: 57. (index mismatch)
I20251215 21:05:54.869897 24043 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20251215 21:05:54.971233 24036 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:54.971375 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 37 FOLLOWER]: Advancing to term 38
I20251215 21:05:54.972247 24036 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.972631 24036 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 38 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:54.973042 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 38 candidate_status { last_received { term: 37 index: 57 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:54.973057 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 38 candidate_status { last_received { term: 37 index: 57 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:54.973423 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:54.973577 24036 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20251215 21:05:54.986052 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 38 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 37. Current term is 38. Ops: []
I20251215 21:05:54.986543 24040 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 37.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251215 21:05:54.986706 24040 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 37 LEADER]: Stepping down as leader of term 37
I20251215 21:05:54.986752 24040 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 37 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:54.986878 24040 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 37.57, Last appended by leader: 57, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:54.987027 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 37 FOLLOWER]: Advancing to term 38
I20251215 21:05:55.050765 24041 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 37 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:55.050894 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 37 FOLLOWER]: Advancing to term 38
I20251215 21:05:55.051785 24041 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.052191 24041 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 38 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:55.052632 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 38 candidate_status { last_received { term: 37 index: 57 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:55.052659 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 38 candidate_status { last_received { term: 37 index: 57 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:55.052845 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 38: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:55.053489 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 38.
I20251215 21:05:55.053738 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:55.053894 24041 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 38 FOLLOWER]: Leader election won for term 38
I20251215 21:05:55.053969 24041 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 38 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:55.054113 24041 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 37.57, Last appended by leader: 57, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.055423 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 37 to 38, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 38 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:55.113945 24036 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:55.114074 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 38 FOLLOWER]: Advancing to term 39
I20251215 21:05:55.114945 24036 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.115357 24036 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 39 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:55.115744 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 39 candidate_status { last_received { term: 37 index: 57 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:55.115744 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 39 candidate_status { last_received { term: 37 index: 57 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:55.115895 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 38 FOLLOWER]: Advancing to term 39
I20251215 21:05:55.116746 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 39.
I20251215 21:05:55.117096 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:55.117302 24036 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 39 FOLLOWER]: Leader election won for term 39
I20251215 21:05:55.117456 24036 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 39 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:55.117595 24036 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 37.57, Last appended by leader: 57, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.118826 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 38 to 39, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 39 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:55.146724 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 39 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 38. Current term is 39. Ops: []
I20251215 21:05:55.147125 24041 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251215 21:05:55.147336 24041 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 38 LEADER]: Stepping down as leader of term 38
I20251215 21:05:55.147387 24041 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:55.147516 24041 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 38.58, Last appended by leader: 58, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.147637 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 38 FOLLOWER]: Advancing to term 39
I20251215 21:05:55.224000 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 39 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 37 index: 57. Preceding OpId from leader: term: 39 index: 58. (index mismatch)
I20251215 21:05:55.224488 24036 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251215 21:05:55.226766 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 39 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 38 index: 58. Preceding OpId from leader: term: 39 index: 58. (term mismatch)
I20251215 21:05:55.226838 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 57
I20251215 21:05:55.226864 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 38.58
I20251215 21:05:55.226888 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:55.227243 24045 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251215 21:05:55.288589 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:55.573366 24040 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 39 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:55.573498 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 39 FOLLOWER]: Advancing to term 40
I20251215 21:05:55.575031 24040 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.575456 24040 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 40 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:55.575809 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 40 candidate_status { last_received { term: 39 index: 58 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:55.575817 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 40 candidate_status { last_received { term: 39 index: 58 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:55.576148 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:55.576297 24040 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20251215 21:05:55.589419 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 40 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 39. Current term is 40. Ops: []
I20251215 21:05:55.589782 24036 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 39.58, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251215 21:05:55.589968 24045 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 39 LEADER]: Stepping down as leader of term 39
I20251215 21:05:55.590034 24045 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:55.590162 24045 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 39.58, Last appended by leader: 58, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.590317 24045 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 39 FOLLOWER]: Advancing to term 40
I20251215 21:05:55.596565 24041 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:55.596649 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 39 FOLLOWER]: Advancing to term 40
I20251215 21:05:55.597419 24041 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.597752 24041 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 40 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:55.598086 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 40 candidate_status { last_received { term: 39 index: 58 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:55.598088 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 40 candidate_status { last_received { term: 39 index: 58 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:55.598234 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 40: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:55.598918 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 40.
I20251215 21:05:55.599169 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:55.599329 24041 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 40 FOLLOWER]: Leader election won for term 40
I20251215 21:05:55.599402 24041 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 40 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:55.599534 24041 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 39.58, Last appended by leader: 58, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.600845 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 39 to 40, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 40 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:55.696786 24040 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:55.696957 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 40 FOLLOWER]: Advancing to term 41
I20251215 21:05:55.698169 24040 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.698661 24040 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 41 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:55.699083 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 41 candidate_status { last_received { term: 39 index: 58 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:55.699083 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 41 candidate_status { last_received { term: 39 index: 58 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:55.699245 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 40 FOLLOWER]: Advancing to term 41
I20251215 21:05:55.700016 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 41.
I20251215 21:05:55.700260 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:55.700470 24040 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 FOLLOWER]: Leader election won for term 41
I20251215 21:05:55.700628 24040 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:55.700784 24040 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 39.58, Last appended by leader: 58, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.702138 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 40 to 41, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 41 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:55.705452 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 40. Current term is 41. Ops: []
I20251215 21:05:55.705781 24041 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251215 21:05:55.705892 24041 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 41.59 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 58
I20251215 21:05:55.705942 24038 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 40 LEADER]: Stepping down as leader of term 40
I20251215 21:05:55.705991 24038 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:55.706099 24038 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 40.59, Last appended by leader: 59, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.706219 24038 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 40 FOLLOWER]: Advancing to term 41
I20251215 21:05:55.777436 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 41 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 40 index: 59. Preceding OpId from leader: term: 41 index: 59. (term mismatch)
I20251215 21:05:55.777537 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 58
I20251215 21:05:55.777566 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 40.59
I20251215 21:05:55.777596 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:55.777997 24040 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251215 21:05:55.780113 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 41 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 39 index: 58. Preceding OpId from leader: term: 41 index: 59. (index mismatch)
I20251215 21:05:55.780498 24040 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251215 21:05:55.972463 24038 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 41 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:55.972597 24038 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 41 FOLLOWER]: Advancing to term 42
I20251215 21:05:55.973497 24038 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.973858 24038 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 42 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:55.974272 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 42 candidate_status { last_received { term: 41 index: 59 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:55.974277 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 42 candidate_status { last_received { term: 41 index: 59 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:55.974648 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:55.974802 24038 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20251215 21:05:55.982012 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 42 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 41. Current term is 42. Ops: []
I20251215 21:05:55.982399 24047 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 41.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251215 21:05:55.982587 24040 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 LEADER]: Stepping down as leader of term 41
I20251215 21:05:55.982645 24040 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:55.982784 24040 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 41.59, Last appended by leader: 59, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:55.982937 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 41 FOLLOWER]: Advancing to term 42
I20251215 21:05:56.045965 24045 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 41 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:56.046101 24045 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 41 FOLLOWER]: Advancing to term 42
I20251215 21:05:56.047060 24045 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.047484 24045 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 42 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:56.047875 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 42 candidate_status { last_received { term: 41 index: 59 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:56.047875 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 42 candidate_status { last_received { term: 41 index: 59 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:56.048064 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 42: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:05:56.048731 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 42.
I20251215 21:05:56.049026 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:56.049201 24045 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 42 FOLLOWER]: Leader election won for term 42
I20251215 21:05:56.049306 24045 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 42 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:56.049446 24045 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 41.59, Last appended by leader: 59, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.050748 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 41 to 42, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 42 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:56.075582 24038 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:56.075706 24038 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 42 FOLLOWER]: Advancing to term 43
I20251215 21:05:56.076572 24038 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.077010 24038 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 43 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:56.077427 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 43 candidate_status { last_received { term: 41 index: 59 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:56.077425 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 43 candidate_status { last_received { term: 41 index: 59 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:56.077548 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 42 FOLLOWER]: Advancing to term 43
I20251215 21:05:56.078305 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 43.
I20251215 21:05:56.078591 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:56.078778 24038 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 43 FOLLOWER]: Leader election won for term 43
I20251215 21:05:56.078856 24038 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 43 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:56.078979 24038 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 41.59, Last appended by leader: 59, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.080220 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 42 to 43, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 43 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:56.134042 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 43 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 42. Current term is 43. Ops: []
I20251215 21:05:56.134541 24045 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251215 21:05:56.134747 24036 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 42 LEADER]: Stepping down as leader of term 42
I20251215 21:05:56.134806 24036 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:56.134922 24036 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 42.60, Last appended by leader: 60, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.135059 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 42 FOLLOWER]: Advancing to term 43
I20251215 21:05:56.172670 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 43 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 42 index: 60. Preceding OpId from leader: term: 43 index: 60. (term mismatch)
I20251215 21:05:56.172755 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 59
I20251215 21:05:56.172772 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 42.60
I20251215 21:05:56.172789 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:56.173182 24038 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251215 21:05:56.175262 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 43 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 41 index: 59. Preceding OpId from leader: term: 43 index: 60. (index mismatch)
I20251215 21:05:56.175694 24048 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251215 21:05:56.276882 24040 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 43 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:56.277010 24040 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 43 FOLLOWER]: Advancing to term 44
I20251215 21:05:56.277849 24040 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.278234 24040 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 44 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:56.278690 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 44 candidate_status { last_received { term: 43 index: 60 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:56.278690 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 44 candidate_status { last_received { term: 43 index: 60 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:56.278842 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 43 FOLLOWER]: Advancing to term 44
I20251215 21:05:56.279626 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 44.
I20251215 21:05:56.279848 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:56.279994 24040 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 FOLLOWER]: Leader election won for term 44
I20251215 21:05:56.280079 24040 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:56.280225 24040 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 43.60, Last appended by leader: 60, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.281759 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 43 to 44, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 44 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:56.288848 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:56.291682 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 43. Current term is 44. Ops: []
I20251215 21:05:56.292068 24038 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 43.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20251215 21:05:56.292219 24038 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 44.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I20251215 21:05:56.292280 24041 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 43 LEADER]: Stepping down as leader of term 43
I20251215 21:05:56.292331 24041 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:56.292450 24041 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 43.60, Last appended by leader: 60, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.292608 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 43 FOLLOWER]: Advancing to term 44
I20251215 21:05:56.292740 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 44 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 43. Current term is 44. Ops: []
I20251215 21:05:56.398656 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 44 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 43 index: 60. Preceding OpId from leader: term: 44 index: 61. (index mismatch)
I20251215 21:05:56.399075 24040 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20251215 21:05:56.401072 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 44 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 43 index: 60. Preceding OpId from leader: term: 44 index: 61. (index mismatch)
I20251215 21:05:56.401612 24040 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20251215 21:05:56.591976 24036 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 44 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:56.592105 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 44 FOLLOWER]: Advancing to term 45
I20251215 21:05:56.592993 24036 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.593402 24036 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 45 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:56.593858 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 45 candidate_status { last_received { term: 44 index: 61 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:56.593858 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 45 candidate_status { last_received { term: 44 index: 61 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:56.594291 24041 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 44 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:56.594377 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 44 FOLLOWER]: Advancing to term 45
I20251215 21:05:56.594352 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:56.594544 24036 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20251215 21:05:56.595150 24041 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.595321 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 45 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 44. Current term is 45. Ops: []
I20251215 21:05:56.595572 24041 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 45 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:56.595701 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 45 candidate_status { last_received { term: 44 index: 61 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:56.595738 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 45 candidate_status { last_received { term: 44 index: 61 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:56.595778 24047 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 44.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251215 21:05:56.595826 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 45: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:56.595953 24047 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 LEADER]: Stepping down as leader of term 44
I20251215 21:05:56.595999 24047 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:56.596076 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:56.596128 24047 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 44.61, Last appended by leader: 61, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.596264 24041 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20251215 21:05:56.596280 24047 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 44 FOLLOWER]: Advancing to term 45
I20251215 21:05:56.726147 24036 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:56.726312 24036 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 45 FOLLOWER]: Advancing to term 46
I20251215 21:05:56.727485 24036 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.727957 24036 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 46 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:56.728367 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 46 candidate_status { last_received { term: 44 index: 61 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:56.728367 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 46 candidate_status { last_received { term: 44 index: 61 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:56.728513 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 45 FOLLOWER]: Advancing to term 46
I20251215 21:05:56.728513 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 45 FOLLOWER]: Advancing to term 46
I20251215 21:05:56.729480 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 46.
I20251215 21:05:56.729480 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 46.
I20251215 21:05:56.729799 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:05:56.729944 24036 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 46 FOLLOWER]: Leader election won for term 46
I20251215 21:05:56.730118 24036 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 46 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:56.730414 24036 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 44.61, Last appended by leader: 61, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.731734 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 44 to 46, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 46 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:56.807938 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 46 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 44 index: 61. Preceding OpId from leader: term: 46 index: 62. (index mismatch)
I20251215 21:05:56.808317 24036 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251215 21:05:56.810257 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 46 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 44 index: 61. Preceding OpId from leader: term: 46 index: 62. (index mismatch)
I20251215 21:05:56.810691 24050 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251215 21:05:56.989187 24047 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 46 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:56.989337 24047 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 46 FOLLOWER]: Advancing to term 47
I20251215 21:05:56.990319 24047 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:56.990759 24047 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 47 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:56.991178 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 47 candidate_status { last_received { term: 46 index: 62 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:56.991178 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 47 candidate_status { last_received { term: 46 index: 62 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:56.991585 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:56.991740 24047 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20251215 21:05:57.002758 24041 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:57.002867 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 46 FOLLOWER]: Advancing to term 47
I20251215 21:05:57.003711 24041 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.004099 24041 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 47 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:57.004448 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 47 candidate_status { last_received { term: 46 index: 62 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:57.004465 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 47 candidate_status { last_received { term: 46 index: 62 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:57.004601 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 47: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:57.004894 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:57.005038 24041 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20251215 21:05:57.013612 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 47 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 46. Current term is 47. Ops: []
I20251215 21:05:57.013988 24051 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 46.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251215 21:05:57.014196 24050 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 46 LEADER]: Stepping down as leader of term 46
I20251215 21:05:57.014266 24050 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:57.014397 24050 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 46.62, Last appended by leader: 62, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.014544 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 46 FOLLOWER]: Advancing to term 47
I20251215 21:05:57.102236 24047 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:57.102380 24047 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 47 FOLLOWER]: Advancing to term 48
I20251215 21:05:57.103274 24047 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.103675 24047 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 48 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:57.104103 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 48 candidate_status { last_received { term: 46 index: 62 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:57.104103 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 48 candidate_status { last_received { term: 46 index: 62 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:57.104251 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 47 FOLLOWER]: Advancing to term 48
I20251215 21:05:57.104287 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 47 FOLLOWER]: Advancing to term 48
I20251215 21:05:57.105198 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 48.
I20251215 21:05:57.105198 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 48.
I20251215 21:05:57.105561 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:05:57.105742 24047 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 48 FOLLOWER]: Leader election won for term 48
I20251215 21:05:57.105831 24047 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 48 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:57.105965 24047 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 46.62, Last appended by leader: 62, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.107282 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 46 to 48, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 48 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:57.186638 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 48 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 46 index: 62. Preceding OpId from leader: term: 48 index: 63. (index mismatch)
I20251215 21:05:57.187023 24047 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251215 21:05:57.189077 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 48 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 46 index: 62. Preceding OpId from leader: term: 48 index: 63. (index mismatch)
I20251215 21:05:57.189417 24049 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251215 21:05:57.289088 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:57.289647 24050 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 48 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:57.289726 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 48 FOLLOWER]: Advancing to term 49
I20251215 21:05:57.290700 24050 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.291076 24050 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 49 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:57.291421 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 49 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:57.291445 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 49 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:57.291850 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:57.292024 24050 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20251215 21:05:57.298370 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 49 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 48. Current term is 49. Ops: []
I20251215 21:05:57.298846 24047 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 48.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20251215 21:05:57.299067 24049 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 48 LEADER]: Stepping down as leader of term 48
I20251215 21:05:57.299130 24049 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 48 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:57.299360 24049 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 48.63, Last appended by leader: 63, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.299556 24049 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 48 FOLLOWER]: Advancing to term 49
I20251215 21:05:57.371850 24041 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 48 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:57.371985 24041 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 48 FOLLOWER]: Advancing to term 49
I20251215 21:05:57.372936 24041 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.373337 24041 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 49 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:57.373754 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 49 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:57.373764 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 49 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:57.373893 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 49: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:57.374670 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 49.
I20251215 21:05:57.374907 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:57.375052 24041 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 49 FOLLOWER]: Leader election won for term 49
I20251215 21:05:57.375207 24041 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 49 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:57.375365 24041 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 48.63, Last appended by leader: 63, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.376631 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 48 to 49, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 49 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:57.421106 24050 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:57.421253 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 49 FOLLOWER]: Advancing to term 50
I20251215 21:05:57.422160 24050 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.422559 24050 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 50 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:57.422966 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 50 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:57.422983 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 50 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:57.423103 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 49 FOLLOWER]: Advancing to term 50
I20251215 21:05:57.423940 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 50.
I20251215 21:05:57.424201 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:57.424364 24050 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 FOLLOWER]: Leader election won for term 50
I20251215 21:05:57.424465 24050 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:57.424607 24050 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 48.63, Last appended by leader: 63, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.426075 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 49 to 50, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 50 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:57.468914 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 49. Current term is 50. Ops: []
I20251215 21:05:57.469386 24041 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20251215 21:05:57.469538 24041 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 50.64 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 63
I20251215 21:05:57.469599 24053 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 49 LEADER]: Stepping down as leader of term 49
I20251215 21:05:57.469648 24053 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 49 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:57.469785 24053 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 49.64, Last appended by leader: 64, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.469918 24053 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 49 FOLLOWER]: Advancing to term 50
I20251215 21:05:57.524497 24049 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:57.524627 24049 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 50 FOLLOWER]: Advancing to term 51
I20251215 21:05:57.525553 24049 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.525959 24049 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 51 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:57.526350 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 51 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:57.526392 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 51 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:57.526541 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 50 FOLLOWER]: Advancing to term 51
I20251215 21:05:57.527318 23751 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b for term 51 because replica has last-logged OpId of term: 49 index: 64, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 63.
I20251215 21:05:57.527601 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:57.527776 24049 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20251215 21:05:57.535269 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 51 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 50. Current term is 51. Ops: []
I20251215 21:05:57.535609 24050 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20251215 21:05:57.535717 24050 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 49.64 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 63
I20251215 21:05:57.535771 24051 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 LEADER]: Stepping down as leader of term 50
I20251215 21:05:57.535836 24051 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:57.535989 24051 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 50.64, Last appended by leader: 64, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.536144 24051 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 50 FOLLOWER]: Advancing to term 51
I20251215 21:05:57.646291 24049 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:57.646420 24049 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 51 FOLLOWER]: Advancing to term 52
I20251215 21:05:57.647333 24049 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.647730 24049 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 52 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:57.648151 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 52 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:57.648155 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 52 candidate_status { last_received { term: 48 index: 63 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:57.648252 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 51 FOLLOWER]: Advancing to term 52
I20251215 21:05:57.648329 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 51 FOLLOWER]: Advancing to term 52
I20251215 21:05:57.649055 23902 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b for term 52 because replica has last-logged OpId of term: 50 index: 64, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 63.
I20251215 21:05:57.649135 23751 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b for term 52 because replica has last-logged OpId of term: 49 index: 64, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 63.
I20251215 21:05:57.649518 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:57.649711 24049 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20251215 21:05:57.731362 24053 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:57.731469 24053 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 52 FOLLOWER]: Advancing to term 53
I20251215 21:05:57.732360 24053 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.732775 24053 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 53 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:57.733206 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 53 candidate_status { last_received { term: 49 index: 64 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:57.733211 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 53 candidate_status { last_received { term: 49 index: 64 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:57.733318 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 52 FOLLOWER]: Advancing to term 53
I20251215 21:05:57.733387 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 52 FOLLOWER]: Advancing to term 53
I20251215 21:05:57.734102 23902 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for term 53 because replica has last-logged OpId of term: 50 index: 64, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 64.
I20251215 21:05:57.734136 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 53.
I20251215 21:05:57.734413 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:57.734573 24053 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 53 FOLLOWER]: Leader election won for term 53
I20251215 21:05:57.734654 24053 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 53 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:57.734789 24053 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 49.64, Last appended by leader: 64, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:57.736079 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 50 to 53, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 53 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:57.818782 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 53 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 50 index: 64. Preceding OpId from leader: term: 53 index: 65. (index mismatch)
I20251215 21:05:57.819200 24053 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20251215 21:05:57.819322 24053 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 50.64 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 63
I20251215 21:05:57.819814 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 63
I20251215 21:05:57.819860 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 50.64
I20251215 21:05:57.819890 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 53 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:57.821491 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 53 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 48 index: 63. Preceding OpId from leader: term: 53 index: 65. (index mismatch)
I20251215 21:05:57.821995 24053 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20251215 21:05:57.998878 24051 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 53 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:57.999024 24051 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 53 FOLLOWER]: Advancing to term 54
I20251215 21:05:58.000011 24051 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.000414 24051 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 54 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.000856 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 54 candidate_status { last_received { term: 53 index: 65 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.000860 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 54 candidate_status { last_received { term: 53 index: 65 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:58.001255 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:58.001418 24051 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20251215 21:05:58.010984 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 54 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 53. Current term is 54. Ops: []
I20251215 21:05:58.011387 24053 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 53.65, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251215 21:05:58.011577 24055 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 53 LEADER]: Stepping down as leader of term 53
I20251215 21:05:58.011636 24055 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 53 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:58.011766 24055 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 53.65, Last appended by leader: 65, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.011914 24055 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 53 FOLLOWER]: Advancing to term 54
I20251215 21:05:58.081347 24049 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 53 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:58.081488 24049 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 53 FOLLOWER]: Advancing to term 54
I20251215 21:05:58.082404 24049 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.082792 24049 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 54 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.083230 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 54 candidate_status { last_received { term: 53 index: 65 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.083252 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 54 candidate_status { last_received { term: 53 index: 65 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:58.083431 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 54: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:58.084088 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 54.
I20251215 21:05:58.084363 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:58.084522 24049 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 54 FOLLOWER]: Leader election won for term 54
I20251215 21:05:58.084612 24049 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 54 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:58.084777 24049 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 53.65, Last appended by leader: 65, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.086184 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 53 to 54, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 54 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:58.108985 24051 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:58.109112 24051 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 54 FOLLOWER]: Advancing to term 55
I20251215 21:05:58.109995 24051 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.110394 24051 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 55 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.110807 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 55 candidate_status { last_received { term: 53 index: 65 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:58.110811 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 55 candidate_status { last_received { term: 53 index: 65 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.110965 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 54 FOLLOWER]: Advancing to term 55
I20251215 21:05:58.111696 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 55.
I20251215 21:05:58.112008 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:58.112190 24051 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 55 FOLLOWER]: Leader election won for term 55
I20251215 21:05:58.112277 24051 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 55 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:58.112414 24051 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 53.65, Last appended by leader: 65, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.113687 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 54 to 55, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 55 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:58.173403 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 55 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 54. Current term is 55. Ops: []
I20251215 21:05:58.173806 24049 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251215 21:05:58.173974 24049 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 54 LEADER]: Stepping down as leader of term 54
I20251215 21:05:58.174018 24049 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 54 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:58.174132 24049 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 54.66, Last appended by leader: 66, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.174255 24049 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 54 FOLLOWER]: Advancing to term 55
I20251215 21:05:58.197858 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 55 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 53 index: 65. Preceding OpId from leader: term: 55 index: 66. (index mismatch)
I20251215 21:05:58.198266 24051 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251215 21:05:58.200277 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 55 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 54 index: 66. Preceding OpId from leader: term: 55 index: 66. (term mismatch)
I20251215 21:05:58.200342 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 65
I20251215 21:05:58.200366 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 54.66
I20251215 21:05:58.200389 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 55 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:58.200750 24057 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251215 21:05:58.289407 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:58.399125 24055 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 55 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:58.399271 24055 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 55 FOLLOWER]: Advancing to term 56
I20251215 21:05:58.400202 24055 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.400597 24055 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 56 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:58.400955 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 56 candidate_status { last_received { term: 55 index: 66 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:58.400983 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 56 candidate_status { last_received { term: 55 index: 66 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:58.401201 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 56 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 55. Current term is 56. Ops: []
I20251215 21:05:58.401363 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:58.401521 24055 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20251215 21:05:58.401698 24050 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 55.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251215 21:05:58.401867 24057 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 55 LEADER]: Stepping down as leader of term 55
I20251215 21:05:58.401921 24057 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:58.402050 24057 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 55.66, Last appended by leader: 66, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.402202 24057 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 55 FOLLOWER]: Advancing to term 56
I20251215 21:05:58.486591 24049 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 55 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:58.486726 24049 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 55 FOLLOWER]: Advancing to term 56
I20251215 21:05:58.487617 24049 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.488052 24049 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 56 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.488482 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 56 candidate_status { last_received { term: 55 index: 66 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.488482 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 56 candidate_status { last_received { term: 55 index: 66 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:58.488662 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 56: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:05:58.489380 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 56.
I20251215 21:05:58.489611 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:58.489748 24049 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 56 FOLLOWER]: Leader election won for term 56
I20251215 21:05:58.489838 24049 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 56 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:58.489990 24049 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 55.66, Last appended by leader: 66, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.491222 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 55 to 56, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 56 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:58.528007 24055 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:58.528131 24055 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 56 FOLLOWER]: Advancing to term 57
I20251215 21:05:58.529073 24055 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.529486 24055 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 57 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:58.530001 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 57 candidate_status { last_received { term: 55 index: 66 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:58.530007 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 57 candidate_status { last_received { term: 55 index: 66 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:58.530124 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 56 FOLLOWER]: Advancing to term 57
I20251215 21:05:58.530937 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 57.
I20251215 21:05:58.531245 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:58.531424 24055 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 FOLLOWER]: Leader election won for term 57
I20251215 21:05:58.531579 24055 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:58.531718 24055 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 55.66, Last appended by leader: 66, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.533113 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 56 to 57, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 57 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:58.573352 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 LEADER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 56. Current term is 57. Ops: []
I20251215 21:05:58.573762 24049 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251215 21:05:58.573899 24049 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 57.67 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 66
I20251215 21:05:58.573956 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 56 LEADER]: Stepping down as leader of term 56
I20251215 21:05:58.574004 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 56 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:58.574138 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 56.67, Last appended by leader: 67, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.574287 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 56 FOLLOWER]: Advancing to term 57
I20251215 21:05:58.631201 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 57 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 55 index: 66. Preceding OpId from leader: term: 57 index: 67. (index mismatch)
I20251215 21:05:58.631613 24055 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251215 21:05:58.633644 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 57 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 56 index: 67. Preceding OpId from leader: term: 57 index: 67. (term mismatch)
I20251215 21:05:58.633848 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 66
I20251215 21:05:58.633877 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 56.67
I20251215 21:05:58.633904 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:58.634224 24058 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251215 21:05:58.734458 24057 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 57 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:58.734583 24057 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 57 FOLLOWER]: Advancing to term 58
I20251215 21:05:58.735564 24057 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.735966 24057 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 58 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.736537 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 58 candidate_status { last_received { term: 57 index: 67 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.736537 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 58 candidate_status { last_received { term: 57 index: 67 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:58.736986 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 58 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:58.737143 24057 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 58 FOLLOWER]: Leader election lost for term 58. Reason: could not achieve majority
I20251215 21:05:58.738830 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 58 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 57. Current term is 58. Ops: []
I20251215 21:05:58.739148 24058 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 57.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251215 21:05:58.739317 24059 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 LEADER]: Stepping down as leader of term 57
I20251215 21:05:58.739374 24059 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:58.739488 24059 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 57.67, Last appended by leader: 67, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.739610 24059 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 57 FOLLOWER]: Advancing to term 58
I20251215 21:05:58.815668 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 57 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:05:58.815795 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 57 FOLLOWER]: Advancing to term 58
I20251215 21:05:58.816668 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.817078 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 58 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.817519 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 58 candidate_status { last_received { term: 57 index: 67 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:58.817519 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 58 candidate_status { last_received { term: 57 index: 67 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.817695 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 58: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:05:58.818440 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 58.
I20251215 21:05:58.818710 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:58.818886 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 58 FOLLOWER]: Leader election won for term 58
I20251215 21:05:58.819005 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 58 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:58.819149 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 57.67, Last appended by leader: 67, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.820465 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 57 to 58, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 58 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:58.870507 24057 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:58.870644 24057 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 58 FOLLOWER]: Advancing to term 59
I20251215 21:05:58.871589 24057 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.871968 24057 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 59 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:58.872476 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 59 candidate_status { last_received { term: 57 index: 67 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:58.872476 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 59 candidate_status { last_received { term: 57 index: 67 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:58.872632 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 58 FOLLOWER]: Advancing to term 59
I20251215 21:05:58.873438 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 59.
I20251215 21:05:58.873715 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:58.873905 24057 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 59 FOLLOWER]: Leader election won for term 59
I20251215 21:05:58.874009 24057 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 59 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:58.874156 24057 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 57.67, Last appended by leader: 67, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.875516 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 58 to 59, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 59 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:58.895830 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 59 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 58. Current term is 59. Ops: []
I20251215 21:05:58.896224 24056 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251215 21:05:58.896389 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 58 LEADER]: Stepping down as leader of term 58
I20251215 21:05:58.896436 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 58 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:58.896567 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 58.68, Last appended by leader: 68, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:58.896697 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 58 FOLLOWER]: Advancing to term 59
I20251215 21:05:58.968736 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 59 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 57 index: 67. Preceding OpId from leader: term: 59 index: 68. (index mismatch)
I20251215 21:05:58.969169 24057 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251215 21:05:58.971117 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 59 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 58 index: 68. Preceding OpId from leader: term: 59 index: 68. (term mismatch)
I20251215 21:05:58.971182 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 67
I20251215 21:05:58.971206 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 58.68
I20251215 21:05:58.971230 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 59 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:58.971554 24057 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251215 21:05:59.237717 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 59 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:59.237835 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 59 FOLLOWER]: Advancing to term 60
I20251215 21:05:59.238720 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.239113 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 60 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:59.239507 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 60 candidate_status { last_received { term: 59 index: 68 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:59.239511 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 60 candidate_status { last_received { term: 59 index: 68 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:59.239866 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:59.240029 24056 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20251215 21:05:59.257717 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 60 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 59. Current term is 60. Ops: []
I20251215 21:05:59.258148 24060 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 59.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251215 21:05:59.258365 24050 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 59 LEADER]: Stepping down as leader of term 59
I20251215 21:05:59.258423 24050 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:59.258553 24050 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 59.68, Last appended by leader: 68, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.258697 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 59 FOLLOWER]: Advancing to term 60
I20251215 21:05:59.289714 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:05:59.334283 24059 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 59 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:05:59.334414 24059 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 59 FOLLOWER]: Advancing to term 60
I20251215 21:05:59.335295 24059 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.335697 24059 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 60 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:59.336091 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 60 candidate_status { last_received { term: 59 index: 68 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:59.336136 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 60 candidate_status { last_received { term: 59 index: 68 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:59.336220 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 60: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:05:59.337023 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 60.
I20251215 21:05:59.337285 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:59.337435 24059 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 60 FOLLOWER]: Leader election won for term 60
I20251215 21:05:59.337589 24059 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 60 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:59.337728 24059 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 59.68, Last appended by leader: 68, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.338951 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 59 to 60, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 60 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:59.349273 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:59.349377 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 60 FOLLOWER]: Advancing to term 61
I20251215 21:05:59.350138 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.350466 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 61 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:59.350800 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 61 candidate_status { last_received { term: 59 index: 68 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:59.350800 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 61 candidate_status { last_received { term: 59 index: 68 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:59.350905 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 60 FOLLOWER]: Advancing to term 61
I20251215 21:05:59.351648 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 61.
I20251215 21:05:59.351881 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:59.352039 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 FOLLOWER]: Leader election won for term 61
I20251215 21:05:59.352120 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:59.352250 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 59.68, Last appended by leader: 68, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.353505 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 60 to 61, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 61 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:59.433275 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 60. Current term is 61. Ops: []
I20251215 21:05:59.433730 24059 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251215 21:05:59.433882 24059 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 61.69 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 68
I20251215 21:05:59.433943 24061 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 60 LEADER]: Stepping down as leader of term 60
I20251215 21:05:59.433993 24061 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 60 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:59.434142 24061 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 60.69, Last appended by leader: 69, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.434283 24061 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 60 FOLLOWER]: Advancing to term 61
I20251215 21:05:59.460785 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 61 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 59 index: 68. Preceding OpId from leader: term: 61 index: 69. (index mismatch)
I20251215 21:05:59.461232 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251215 21:05:59.463403 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 61 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 60 index: 69. Preceding OpId from leader: term: 61 index: 69. (term mismatch)
I20251215 21:05:59.463467 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 68
I20251215 21:05:59.463491 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 60.69
I20251215 21:05:59.463514 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 61 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:59.464151 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251215 21:05:59.689765 24061 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 61 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:59.689903 24061 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 61 FOLLOWER]: Advancing to term 62
I20251215 21:05:59.690790 24061 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.691175 24061 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 62 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:59.691589 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 62 candidate_status { last_received { term: 61 index: 69 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:59.691587 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 62 candidate_status { last_received { term: 61 index: 69 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:59.691975 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:05:59.692139 24061 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20251215 21:05:59.697731 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 62 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 61. Current term is 62. Ops: []
I20251215 21:05:59.698098 24062 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 61.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251215 21:05:59.698289 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 LEADER]: Stepping down as leader of term 61
I20251215 21:05:59.698345 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:05:59.698488 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 61.69, Last appended by leader: 69, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.698634 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 61 FOLLOWER]: Advancing to term 62
I20251215 21:05:59.758742 24050 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 61 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:05:59.758870 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 61 FOLLOWER]: Advancing to term 62
I20251215 21:05:59.759739 24050 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.760149 24050 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 62 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:05:59.760573 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 62 candidate_status { last_received { term: 61 index: 69 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:05:59.760576 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 62 candidate_status { last_received { term: 61 index: 69 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:59.760703 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 62: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:05:59.761518 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 62.
I20251215 21:05:59.761849 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:05:59.762029 24050 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 62 FOLLOWER]: Leader election won for term 62
I20251215 21:05:59.762190 24050 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 62 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:59.762336 24050 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 61.69, Last appended by leader: 69, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.763649 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 61 to 62, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 62 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:05:59.830335 24061 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 62 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:05:59.830461 24061 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 62 FOLLOWER]: Advancing to term 63
I20251215 21:05:59.831346 24061 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.831725 24061 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 63 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:05:59.832147 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 63 candidate_status { last_received { term: 61 index: 69 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:05:59.832145 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 63 candidate_status { last_received { term: 61 index: 69 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:05:59.832291 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 62 FOLLOWER]: Advancing to term 63
I20251215 21:05:59.833072 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 63.
I20251215 21:05:59.833303 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:05:59.833451 24061 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 FOLLOWER]: Leader election won for term 63
I20251215 21:05:59.833535 24061 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:05:59.833667 24061 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 61.69, Last appended by leader: 69, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.834949 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 62 to 63, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 63 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:05:59.847410 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 62. Current term is 63. Ops: []
I20251215 21:05:59.847744 24050 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251215 21:05:59.847854 24050 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 63.70 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69
I20251215 21:05:59.847904 24063 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 62 LEADER]: Stepping down as leader of term 62
I20251215 21:05:59.847966 24063 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 62 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:05:59.848093 24063 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 62.70, Last appended by leader: 70, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:05:59.848238 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 62 FOLLOWER]: Advancing to term 63
I20251215 21:05:59.913137 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 63 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 61 index: 69. Preceding OpId from leader: term: 63 index: 70. (index mismatch)
I20251215 21:05:59.913516 24061 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251215 21:05:59.915421 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 63 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 62 index: 70. Preceding OpId from leader: term: 63 index: 70. (term mismatch)
I20251215 21:05:59.915485 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 69
I20251215 21:05:59.915510 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 62.70
I20251215 21:05:59.915534 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 63 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:05:59.915853 24061 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251215 21:06:00.016667 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 63 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:00.016830 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 63 FOLLOWER]: Advancing to term 64
I20251215 21:06:00.017751 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.018172 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 64 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:00.018594 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 64 candidate_status { last_received { term: 63 index: 70 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:00.018600 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 64 candidate_status { last_received { term: 63 index: 70 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:00.018970 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:00.019126 24056 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20251215 21:06:00.032825 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 64 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 63. Current term is 64. Ops: []
I20251215 21:06:00.033238 24061 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 63.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251215 21:06:00.033429 24064 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 LEADER]: Stepping down as leader of term 63
I20251215 21:06:00.033484 24064 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:00.033600 24064 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 63.70, Last appended by leader: 70, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.033732 24064 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 63 FOLLOWER]: Advancing to term 64
I20251215 21:06:00.096278 24063 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 63 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:00.096411 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 63 FOLLOWER]: Advancing to term 64
I20251215 21:06:00.097369 24063 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.097769 24063 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 64 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:00.098176 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 64 candidate_status { last_received { term: 63 index: 70 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:00.098176 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 64 candidate_status { last_received { term: 63 index: 70 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:00.098349 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 64 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 64: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:00.099061 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 64.
I20251215 21:06:00.099359 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:00.099543 24063 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 64 FOLLOWER]: Leader election won for term 64
I20251215 21:06:00.099624 24063 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 64 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:00.099761 24063 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 63.70, Last appended by leader: 70, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.101068 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 63 to 64, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 64 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:00.145620 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:00.145751 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 64 FOLLOWER]: Advancing to term 65
I20251215 21:06:00.146637 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.147037 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 65 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:00.147460 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 65 candidate_status { last_received { term: 63 index: 70 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:00.147469 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 65 candidate_status { last_received { term: 63 index: 70 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:00.147612 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 64 FOLLOWER]: Advancing to term 65
I20251215 21:06:00.148360 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 65.
I20251215 21:06:00.148610 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:00.148772 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 FOLLOWER]: Leader election won for term 65
I20251215 21:06:00.148883 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:00.149024 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 63.70, Last appended by leader: 70, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.150332 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 64 to 65, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 65 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:00.198045 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 64. Current term is 65. Ops: []
I20251215 21:06:00.198473 24063 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251215 21:06:00.198609 24063 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 65.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 70
I20251215 21:06:00.198663 24050 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 64 LEADER]: Stepping down as leader of term 64
I20251215 21:06:00.198709 24050 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 64 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:00.198832 24050 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 64.71, Last appended by leader: 71, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.198966 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 64 FOLLOWER]: Advancing to term 65
I20251215 21:06:00.252575 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 65 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 63 index: 70. Preceding OpId from leader: term: 65 index: 71. (index mismatch)
I20251215 21:06:00.252993 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251215 21:06:00.254966 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 65 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 64 index: 71. Preceding OpId from leader: term: 65 index: 71. (term mismatch)
I20251215 21:06:00.255028 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 70
I20251215 21:06:00.255053 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 64.71
I20251215 21:06:00.255077 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 65 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:00.255425 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251215 21:06:00.290035 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:00.471733 24050 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 65 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:00.471886 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 65 FOLLOWER]: Advancing to term 66
I20251215 21:06:00.473706 24050 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.474511 24050 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 66 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:00.475128 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 66 candidate_status { last_received { term: 65 index: 71 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:00.475602 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 66 candidate_status { last_received { term: 65 index: 71 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:00.476076 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:00.476431 24050 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20251215 21:06:00.479972 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 66 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 65. Current term is 66. Ops: []
I20251215 21:06:00.480438 24056 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 65.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251215 21:06:00.480676 24062 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 LEADER]: Stepping down as leader of term 65
I20251215 21:06:00.480756 24062 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:00.480940 24062 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 65.71, Last appended by leader: 71, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.481143 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 65 FOLLOWER]: Advancing to term 66
I20251215 21:06:00.554848 24064 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 65 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:00.555006 24064 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 65 FOLLOWER]: Advancing to term 66
I20251215 21:06:00.556000 24064 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.556442 24064 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 66 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:00.556856 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 66 candidate_status { last_received { term: 65 index: 71 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:00.556883 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 66 candidate_status { last_received { term: 65 index: 71 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:00.557025 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 66: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:00.557749 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 66 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 66.
I20251215 21:06:00.558022 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 66 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:00.558207 24064 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 66 FOLLOWER]: Leader election won for term 66
I20251215 21:06:00.558362 24064 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 66 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:00.558501 24064 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 65.71, Last appended by leader: 71, Current term: 66, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.559909 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 65 to 66, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 66 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:00.587957 24050 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:00.588096 24050 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 66 FOLLOWER]: Advancing to term 67
I20251215 21:06:00.589730 24050 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.590204 24050 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 67 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:00.590590 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 67 candidate_status { last_received { term: 65 index: 71 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:00.590595 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 67 candidate_status { last_received { term: 65 index: 71 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:00.590692 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 66 FOLLOWER]: Advancing to term 67
I20251215 21:06:00.591434 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 67.
I20251215 21:06:00.591693 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:00.591868 24050 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 FOLLOWER]: Leader election won for term 67
I20251215 21:06:00.591967 24050 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:00.592121 24050 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 65.71, Last appended by leader: 71, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.593778 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 66 to 67, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 67 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:00.665120 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 66. Current term is 67. Ops: []
I20251215 21:06:00.665532 24064 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251215 21:06:00.665668 24064 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 67.72 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 71
I20251215 21:06:00.665725 24066 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 66 LEADER]: Stepping down as leader of term 66
I20251215 21:06:00.665776 24066 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 66 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:00.665896 24066 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 66.72, Last appended by leader: 72, Current term: 66, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.666070 24066 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 66 FOLLOWER]: Advancing to term 67
I20251215 21:06:00.683305 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 67 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 65 index: 71. Preceding OpId from leader: term: 67 index: 72. (index mismatch)
I20251215 21:06:00.683633 24050 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251215 21:06:00.685649 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 67 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 66 index: 72. Preceding OpId from leader: term: 67 index: 72. (term mismatch)
I20251215 21:06:00.685710 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 71
I20251215 21:06:00.685732 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 66.72
I20251215 21:06:00.685753 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 67 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:00.686105 24067 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251215 21:06:00.786587 24062 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 67 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:00.786713 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 67 FOLLOWER]: Advancing to term 68
I20251215 21:06:00.787835 24062 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.788270 24062 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 68 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:00.788764 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 68 candidate_status { last_received { term: 67 index: 72 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:00.788842 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 68 candidate_status { last_received { term: 67 index: 72 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:00.788928 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 67 FOLLOWER]: Advancing to term 68
I20251215 21:06:00.789932 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 68.
I20251215 21:06:00.790225 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:00.790377 24062 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 FOLLOWER]: Leader election won for term 68
I20251215 21:06:00.790464 24062 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:00.790607 24062 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 67.72, Last appended by leader: 72, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.792147 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 67 to 68, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 68 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:00.800920 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 67. Current term is 68. Ops: []
I20251215 21:06:00.801236 24067 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 67.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20251215 21:06:00.801354 24067 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 68.73 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 72
I20251215 21:06:00.801404 24063 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 LEADER]: Stepping down as leader of term 67
I20251215 21:06:00.801452 24063 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:00.801551 24063 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 67.72, Last appended by leader: 72, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:00.801666 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 67 FOLLOWER]: Advancing to term 68
I20251215 21:06:00.890169 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 68 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 67 index: 72. Preceding OpId from leader: term: 68 index: 73. (index mismatch)
I20251215 21:06:00.890535 24062 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20251215 21:06:00.892670 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 68 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 67 index: 72. Preceding OpId from leader: term: 68 index: 73. (index mismatch)
I20251215 21:06:00.893128 24068 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20251215 21:06:01.081010 24066 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 68 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:01.081140 24066 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 68 FOLLOWER]: Advancing to term 69
I20251215 21:06:01.082044 24066 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.082412 24066 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 69 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:01.082805 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 69 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.082808 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 69 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.083168 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:01.083324 24066 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I20251215 21:06:01.091985 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 69 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 68. Current term is 69. Ops: []
I20251215 21:06:01.092348 24068 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 68.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20251215 21:06:01.092535 24062 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 LEADER]: Stepping down as leader of term 68
I20251215 21:06:01.092594 24062 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:01.092727 24062 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 68.73, Last appended by leader: 73, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.092904 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 68 FOLLOWER]: Advancing to term 69
I20251215 21:06:01.113487 24063 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 68 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:01.113624 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 68 FOLLOWER]: Advancing to term 69
I20251215 21:06:01.114500 24063 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.114881 24063 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 69 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:01.115290 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 69 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.115319 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 69 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:01.115463 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 69: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:01.116060 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 69.
I20251215 21:06:01.116349 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:01.116528 24063 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 69 FOLLOWER]: Leader election won for term 69
I20251215 21:06:01.116616 24063 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 69 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:01.116756 24063 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 68.73, Last appended by leader: 73, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.118103 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 68 to 69, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 69 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:01.207417 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 69 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 68 index: 73. Preceding OpId from leader: term: 69 index: 74. (index mismatch)
I20251215 21:06:01.207839 24063 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20251215 21:06:01.208923 24066 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:01.209069 24066 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 69 FOLLOWER]: Advancing to term 70
I20251215 21:06:01.210321 24066 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.210563 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 70 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 69. Current term is 70. Ops: []
I20251215 21:06:01.211046 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 70 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.211170 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 70 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.211323 24063 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20251215 21:06:01.211463 24063 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 69 LEADER]: Stepping down as leader of term 69
I20251215 21:06:01.211488 24066 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 70 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:01.211474 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 70 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:01.211544 24063 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 69 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:01.211632 24066 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 70 FOLLOWER]: Leader election lost for term 70. Reason: could not achieve majority
I20251215 21:06:01.211670 24063 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 69.74, Last appended by leader: 74, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.211807 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 69 FOLLOWER]: Advancing to term 70
I20251215 21:06:01.290337 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:01.318310 24062 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 69 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:01.318447 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 69 FOLLOWER]: Advancing to term 70
I20251215 21:06:01.319324 24062 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.319717 24062 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 70 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:01.320155 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 70 candidate_status { last_received { term: 69 index: 74 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:01.320153 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 70 candidate_status { last_received { term: 69 index: 74 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.320330 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 70 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 70: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:01.321055 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 70.
I20251215 21:06:01.321297 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:01.321447 24062 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 70 FOLLOWER]: Leader election won for term 70
I20251215 21:06:01.321529 24062 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 70 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:01.321666 24062 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 69.74, Last appended by leader: 74, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.322890 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 69 to 70, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 70 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:01.360227 24066 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 70 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:01.360365 24066 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 70 FOLLOWER]: Advancing to term 71
I20251215 21:06:01.361299 24066 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.361676 24066 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 71 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:01.362084 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 71 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.362084 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 71 candidate_status { last_received { term: 68 index: 73 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.362229 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 70 FOLLOWER]: Advancing to term 71
I20251215 21:06:01.362934 23902 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 71 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for term 71 because replica has last-logged OpId of term: 69 index: 74, which is greater than that of the candidate, which has last-logged OpId of term: 68 index: 73.
I20251215 21:06:01.363169 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 71 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:01.363317 24066 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 71 FOLLOWER]: Leader election lost for term 71. Reason: could not achieve majority
I20251215 21:06:01.407933 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 71 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 70. Current term is 71. Ops: []
I20251215 21:06:01.407933 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 71 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 70. Current term is 71. Ops: []
I20251215 21:06:01.408528 24068 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20251215 21:06:01.408752 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 70 LEADER]: Stepping down as leader of term 70
I20251215 21:06:01.408846 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 70 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:01.408761 24062 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20251215 21:06:01.409008 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 70.75, Last appended by leader: 75, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.409157 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 70 FOLLOWER]: Advancing to term 71
I20251215 21:06:01.437578 24063 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 71 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:01.437706 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 71 FOLLOWER]: Advancing to term 72
I20251215 21:06:01.438580 24063 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.438982 24063 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 72 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:01.439378 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 72 candidate_status { last_received { term: 69 index: 74 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.439374 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 72 candidate_status { last_received { term: 69 index: 74 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:01.439494 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 71 FOLLOWER]: Advancing to term 72
I20251215 21:06:01.439499 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 71 FOLLOWER]: Advancing to term 72
I20251215 21:06:01.440217 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 72.
I20251215 21:06:01.440256 23826 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for term 72 because replica has last-logged OpId of term: 70 index: 75, which is greater than that of the candidate, which has last-logged OpId of term: 69 index: 74.
I20251215 21:06:01.440485 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters:
I20251215 21:06:01.440631 24063 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 72 FOLLOWER]: Leader election won for term 72
I20251215 21:06:01.440706 24063 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 72 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:01.440864 24063 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 69.74, Last appended by leader: 74, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.442114 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 70 to 72, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 72 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:01.543352 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 72 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 70 index: 75. Preceding OpId from leader: term: 72 index: 75. (term mismatch)
I20251215 21:06:01.543438 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 74
I20251215 21:06:01.543455 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 70.75
I20251215 21:06:01.543473 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 72 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:01.543810 24063 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20251215 21:06:01.545858 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 72 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 68 index: 73. Preceding OpId from leader: term: 72 index: 75. (index mismatch)
I20251215 21:06:01.546192 24069 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
I20251215 21:06:01.647311 24070 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 72 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:01.647433 24070 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 72 FOLLOWER]: Advancing to term 73
I20251215 21:06:01.648343 24070 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.648746 24070 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 73 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:01.649122 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 73 candidate_status { last_received { term: 72 index: 75 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.649122 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 73 candidate_status { last_received { term: 72 index: 75 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.649531 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 73 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:01.649703 24070 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 73 FOLLOWER]: Leader election lost for term 73. Reason: could not achieve majority
I20251215 21:06:01.658743 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 73 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 72. Current term is 73. Ops: []
I20251215 21:06:01.659102 24063 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 72.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251215 21:06:01.659282 24063 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 72 LEADER]: Stepping down as leader of term 72
I20251215 21:06:01.659329 24063 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 72 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:01.659456 24063 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 72.75, Last appended by leader: 75, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.659612 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 72 FOLLOWER]: Advancing to term 73
I20251215 21:06:01.741598 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 72 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:01.741724 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 72 FOLLOWER]: Advancing to term 73
I20251215 21:06:01.742631 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.743016 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 73 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:01.743412 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 73 candidate_status { last_received { term: 72 index: 75 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:01.743420 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 73 candidate_status { last_received { term: 72 index: 75 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.743559 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 73: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:01.744277 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 73.
I20251215 21:06:01.744513 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:01.744679 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 73 FOLLOWER]: Leader election won for term 73
I20251215 21:06:01.744763 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 73 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:01.744925 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 72.75, Last appended by leader: 75, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.746276 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 72 to 73, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 73 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:01.785351 24070 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:01.785483 24070 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 73 FOLLOWER]: Advancing to term 74
I20251215 21:06:01.786422 24070 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.786808 24070 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 74 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:01.787202 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 74 candidate_status { last_received { term: 72 index: 75 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.787202 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 74 candidate_status { last_received { term: 72 index: 75 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:01.787362 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 73 FOLLOWER]: Advancing to term 74
I20251215 21:06:01.788115 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 74.
I20251215 21:06:01.788360 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:01.788511 24070 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 74 FOLLOWER]: Leader election won for term 74
I20251215 21:06:01.788590 24070 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 74 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:01.788720 24070 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 72.75, Last appended by leader: 75, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.790115 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 73 to 74, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 74 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:01.827369 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 74 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 73. Current term is 74. Ops: []
I20251215 21:06:01.827806 24056 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251215 21:06:01.827979 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 73 LEADER]: Stepping down as leader of term 73
I20251215 21:06:01.828025 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 73 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:01.828151 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 73.76, Last appended by leader: 76, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.828282 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 73 FOLLOWER]: Advancing to term 74
I20251215 21:06:01.871256 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 74 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 72 index: 75. Preceding OpId from leader: term: 74 index: 76. (index mismatch)
I20251215 21:06:01.871680 24070 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251215 21:06:01.872042 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 74 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 73 index: 76. Preceding OpId from leader: term: 74 index: 76. (term mismatch)
I20251215 21:06:01.872105 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 75
I20251215 21:06:01.872130 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 73.76
I20251215 21:06:01.872160 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 74 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:01.872576 24070 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251215 21:06:01.974113 24063 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 74 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:01.974238 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 74 FOLLOWER]: Advancing to term 75
I20251215 21:06:01.975095 24063 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.975476 24063 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 75 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:01.975894 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 75 candidate_status { last_received { term: 74 index: 76 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:01.975894 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 75 candidate_status { last_received { term: 74 index: 76 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:01.976289 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:01.976452 24063 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20251215 21:06:01.998838 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 75 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 74. Current term is 75. Ops: []
I20251215 21:06:01.999250 24070 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 74.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251215 21:06:01.999442 24072 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 74 LEADER]: Stepping down as leader of term 74
I20251215 21:06:01.999501 24072 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 74 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:01.999629 24072 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 74.76, Last appended by leader: 76, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:01.999792 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 74 FOLLOWER]: Advancing to term 75
I20251215 21:06:02.069990 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 74 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:02.070122 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 74 FOLLOWER]: Advancing to term 75
I20251215 21:06:02.070978 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.071374 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 75 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:02.071808 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 75 candidate_status { last_received { term: 74 index: 76 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:02.071808 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 75 candidate_status { last_received { term: 74 index: 76 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:02.071954 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 75: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:02.072666 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 75 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 75.
I20251215 21:06:02.072937 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 75 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:02.073094 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 75 FOLLOWER]: Leader election won for term 75
I20251215 21:06:02.073191 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 75 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:02.073343 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 74.76, Last appended by leader: 76, Current term: 75, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.074615 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 74 to 75, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 75 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:02.094408 24063 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 75 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:02.094523 24063 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 75 FOLLOWER]: Advancing to term 76
I20251215 21:06:02.095340 24063 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.095708 24063 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 76 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:02.096040 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 76 candidate_status { last_received { term: 74 index: 76 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:02.096040 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 76 candidate_status { last_received { term: 74 index: 76 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:02.096181 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 75 FOLLOWER]: Advancing to term 76
I20251215 21:06:02.096905 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 76.
I20251215 21:06:02.097158 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 76 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:02.097304 24063 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 76 FOLLOWER]: Leader election won for term 76
I20251215 21:06:02.097383 24063 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 76 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:02.097514 24063 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 74.76, Last appended by leader: 76, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.098703 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 75 to 76, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 76 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:02.158797 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 76 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 75. Current term is 76. Ops: []
I20251215 21:06:02.159245 24056 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251215 21:06:02.159425 24062 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 75 LEADER]: Stepping down as leader of term 75
I20251215 21:06:02.159482 24062 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 75 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:02.159615 24062 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 75.77, Last appended by leader: 77, Current term: 75, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.159755 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 75 FOLLOWER]: Advancing to term 76
I20251215 21:06:02.176964 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 76 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 75 index: 77. Preceding OpId from leader: term: 76 index: 77. (term mismatch)
I20251215 21:06:02.177043 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 76
I20251215 21:06:02.177060 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 75.77
I20251215 21:06:02.177078 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 76 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:02.177429 24063 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251215 21:06:02.179370 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 76 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 74 index: 76. Preceding OpId from leader: term: 76 index: 77. (index mismatch)
I20251215 21:06:02.179699 24069 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251215 21:06:02.290647 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:02.370877 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 76 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:02.371024 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 76 FOLLOWER]: Advancing to term 77
I20251215 21:06:02.371908 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.372390 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 77 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:02.372816 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 77 candidate_status { last_received { term: 76 index: 77 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:02.372821 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 77 candidate_status { last_received { term: 76 index: 77 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:02.373193 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 77 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:02.373337 24072 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 77 FOLLOWER]: Leader election lost for term 77. Reason: could not achieve majority
I20251215 21:06:02.382725 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 77 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 76. Current term is 77. Ops: []
I20251215 21:06:02.383101 24069 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 76.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251215 21:06:02.383292 24073 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 76 LEADER]: Stepping down as leader of term 76
I20251215 21:06:02.383350 24073 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 76 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:02.383488 24073 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 76.77, Last appended by leader: 77, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.383638 24073 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 76 FOLLOWER]: Advancing to term 77
I20251215 21:06:02.395305 24062 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 76 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:02.395401 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 76 FOLLOWER]: Advancing to term 77
I20251215 21:06:02.396186 24062 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.396533 24062 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 77 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:02.396912 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 77 candidate_status { last_received { term: 76 index: 77 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:02.396989 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 77 candidate_status { last_received { term: 76 index: 77 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:02.397053 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 77 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 77: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:02.397780 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 77.
I20251215 21:06:02.397987 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 77 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:02.398123 24062 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 77 FOLLOWER]: Leader election won for term 77
I20251215 21:06:02.398201 24062 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 77 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:02.398337 24062 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 76.77, Last appended by leader: 77, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.399740 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 76 to 77, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 77 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:02.486613 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 77 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:02.486745 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 77 FOLLOWER]: Advancing to term 78
I20251215 21:06:02.487699 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.488087 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 78 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:02.488521 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 78 candidate_status { last_received { term: 76 index: 77 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:02.488519 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 78 candidate_status { last_received { term: 76 index: 77 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:02.488670 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 77 FOLLOWER]: Advancing to term 78
I20251215 21:06:02.489455 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 78.
I20251215 21:06:02.489698 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 78 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:02.489847 24072 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 FOLLOWER]: Leader election won for term 78
I20251215 21:06:02.489943 24072 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:02.490078 24072 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 76.77, Last appended by leader: 77, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.491384 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 77 to 78, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 78 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:02.494468 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 LEADER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 77. Current term is 78. Ops: []
I20251215 21:06:02.494778 24062 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251215 21:06:02.494881 24062 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 78.78 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 77
I20251215 21:06:02.494930 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 77 LEADER]: Stepping down as leader of term 77
I20251215 21:06:02.494979 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 77 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:02.495080 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 77.78, Last appended by leader: 78, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.495191 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 77 FOLLOWER]: Advancing to term 78
I20251215 21:06:02.577540 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 78 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 76 index: 77. Preceding OpId from leader: term: 78 index: 78. (index mismatch)
I20251215 21:06:02.577936 24072 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251215 21:06:02.580184 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 78 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 77 index: 78. Preceding OpId from leader: term: 78 index: 78. (term mismatch)
I20251215 21:06:02.580340 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 77
I20251215 21:06:02.580375 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 77.78
I20251215 21:06:02.580452 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 78 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:02.580837 24070 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251215 21:06:02.763856 24073 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 78 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:02.763984 24073 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 78 FOLLOWER]: Advancing to term 79
I20251215 21:06:02.764849 24073 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.765297 24073 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 79 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:02.765646 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 79 candidate_status { last_received { term: 78 index: 78 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:02.765651 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 79 candidate_status { last_received { term: 78 index: 78 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:02.766026 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:02.766177 24073 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20251215 21:06:02.783914 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 79 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 78. Current term is 79. Ops: []
I20251215 21:06:02.784317 24070 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 78.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251215 21:06:02.784518 24072 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 LEADER]: Stepping down as leader of term 78
I20251215 21:06:02.784575 24072 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:02.784690 24072 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 78.78, Last appended by leader: 78, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.784849 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 78 FOLLOWER]: Advancing to term 79
I20251215 21:06:02.847473 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 78 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:02.847606 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 78 FOLLOWER]: Advancing to term 79
I20251215 21:06:02.848481 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.849013 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 79 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:02.849432 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 79 candidate_status { last_received { term: 78 index: 78 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:02.849436 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 79 candidate_status { last_received { term: 78 index: 78 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:02.849558 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 79: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:02.850371 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 79.
I20251215 21:06:02.850653 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:02.850809 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 79 FOLLOWER]: Leader election won for term 79
I20251215 21:06:02.850885 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 79 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:02.851027 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 78.78, Last appended by leader: 78, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.852375 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 78 to 79, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 79 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:02.875020 24073 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:02.875145 24073 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 79 FOLLOWER]: Advancing to term 80
I20251215 21:06:02.875988 24073 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.876381 24073 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 80 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:02.876731 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 80 candidate_status { last_received { term: 78 index: 78 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:02.876731 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 80 candidate_status { last_received { term: 78 index: 78 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:02.876892 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 79 FOLLOWER]: Advancing to term 80
I20251215 21:06:02.877642 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 80.
I20251215 21:06:02.877913 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:02.878113 24073 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 FOLLOWER]: Leader election won for term 80
I20251215 21:06:02.878216 24073 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:02.878362 24073 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 78.78, Last appended by leader: 78, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.879664 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 79 to 80, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 80 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:02.935539 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 LEADER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 79. Current term is 80. Ops: []
I20251215 21:06:02.935966 24056 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251215 21:06:02.936100 24056 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 80.79 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 78
I20251215 21:06:02.936156 24062 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 79 LEADER]: Stepping down as leader of term 79
I20251215 21:06:02.936205 24062 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 79 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:02.936327 24062 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 79.79, Last appended by leader: 79, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:02.936450 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 79 FOLLOWER]: Advancing to term 80
I20251215 21:06:02.993762 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 80 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 79 index: 79. Preceding OpId from leader: term: 80 index: 79. (term mismatch)
I20251215 21:06:02.993847 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 78
I20251215 21:06:02.993865 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 79.79
I20251215 21:06:02.993882 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 80 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:02.994225 24073 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251215 21:06:02.996301 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 80 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 78 index: 78. Preceding OpId from leader: term: 80 index: 79. (index mismatch)
I20251215 21:06:02.996879 24075 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251215 21:06:03.180620 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 80 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:03.180747 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 80 FOLLOWER]: Advancing to term 81
I20251215 21:06:03.181635 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.181867 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 80. Current term is 81. Ops: []
I20251215 21:06:03.182080 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 81 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:03.182276 24062 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 80 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:03.182348 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 81 candidate_status { last_received { term: 80 index: 79 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:03.182363 24062 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 80 FOLLOWER]: Advancing to term 81
I20251215 21:06:03.182348 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 81 candidate_status { last_received { term: 80 index: 79 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:03.182600 24075 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 80.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251215 21:06:03.182801 24075 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 LEADER]: Stepping down as leader of term 80
I20251215 21:06:03.182855 24075 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:03.183003 24075 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 80.79, Last appended by leader: 79, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.183148 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 80 FOLLOWER]: Advancing to term 81
I20251215 21:06:03.183336 24062 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.183513 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 81: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:03.183760 24062 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 81 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:03.183889 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:03.184044 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 81 candidate_status { last_received { term: 80 index: 79 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:03.184062 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 81 candidate_status { last_received { term: 80 index: 79 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:03.184123 24072 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20251215 21:06:03.184278 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 81: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:03.184835 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 81.
I20251215 21:06:03.185071 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:03.185223 24062 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 FOLLOWER]: Leader election won for term 81
I20251215 21:06:03.185303 24062 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:03.185423 24062 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 80.79, Last appended by leader: 79, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.186641 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 80 to 81, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 81 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:03.261870 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 80 index: 79. Preceding OpId from leader: term: 81 index: 80. (index mismatch)
I20251215 21:06:03.262306 24062 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251215 21:06:03.264237 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 81 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 80 index: 79. Preceding OpId from leader: term: 81 index: 80. (index mismatch)
I20251215 21:06:03.264529 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251215 21:06:03.290925 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:03.455587 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:03.455706 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 81 FOLLOWER]: Advancing to term 82
I20251215 21:06:03.456552 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.456936 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 82 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:03.457342 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 82 candidate_status { last_received { term: 81 index: 80 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:03.457342 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 82 candidate_status { last_received { term: 81 index: 80 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:03.457769 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 82 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:03.457919 24072 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 82 FOLLOWER]: Leader election lost for term 82. Reason: could not achieve majority
I20251215 21:06:03.462409 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 82 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 81. Current term is 82. Ops: []
I20251215 21:06:03.462731 24076 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 81.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251215 21:06:03.462891 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 LEADER]: Stepping down as leader of term 81
I20251215 21:06:03.462944 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:03.463055 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 81.80, Last appended by leader: 80, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.463164 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 81 FOLLOWER]: Advancing to term 82
I20251215 21:06:03.545053 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 81 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:03.545179 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 81 FOLLOWER]: Advancing to term 82
I20251215 21:06:03.546082 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.546494 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 82 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:03.546840 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 82 candidate_status { last_received { term: 81 index: 80 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:03.546989 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 82 candidate_status { last_received { term: 81 index: 80 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:03.547147 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 82 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 82: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:03.547663 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 82.
I20251215 21:06:03.547906 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:03.548076 24075 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 82 FOLLOWER]: Leader election won for term 82
I20251215 21:06:03.548174 24075 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 82 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:03.548332 24075 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 81.80, Last appended by leader: 80, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.549721 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 81 to 82, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 82 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:03.591361 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 82 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:03.591482 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 82 FOLLOWER]: Advancing to term 83
I20251215 21:06:03.592331 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.592695 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 83 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:03.593109 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 83 candidate_status { last_received { term: 81 index: 80 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:03.593106 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 83 candidate_status { last_received { term: 81 index: 80 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:03.593240 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 82 FOLLOWER]: Advancing to term 83
I20251215 21:06:03.593946 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 83.
I20251215 21:06:03.594265 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:03.594434 24072 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 FOLLOWER]: Leader election won for term 83
I20251215 21:06:03.594596 24072 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:03.594733 24072 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 81.80, Last appended by leader: 80, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.595981 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 82 to 83, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 83 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:03.627434 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 82. Current term is 83. Ops: []
I20251215 21:06:03.627803 24075 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251215 21:06:03.627936 24075 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 83.81 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 80
I20251215 21:06:03.627993 24073 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 82 LEADER]: Stepping down as leader of term 82
I20251215 21:06:03.628065 24073 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:03.628213 24073 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 82.81, Last appended by leader: 81, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.628357 24073 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 82 FOLLOWER]: Advancing to term 83
I20251215 21:06:03.671207 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 83 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 82 index: 81. Preceding OpId from leader: term: 83 index: 81. (term mismatch)
I20251215 21:06:03.671299 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 80
I20251215 21:06:03.671316 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 82.81
I20251215 21:06:03.671334 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 83 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:03.671669 24072 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251215 21:06:03.673588 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 83 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 81 index: 80. Preceding OpId from leader: term: 83 index: 81. (index mismatch)
I20251215 21:06:03.673928 24078 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251215 21:06:03.775672 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 83 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:03.775790 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 83 FOLLOWER]: Advancing to term 84
I20251215 21:06:03.776615 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.776988 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 84 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:03.777356 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 84 candidate_status { last_received { term: 83 index: 81 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:03.777356 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 84 candidate_status { last_received { term: 83 index: 81 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:03.777736 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 84 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:03.777891 24056 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 84 FOLLOWER]: Leader election lost for term 84. Reason: could not achieve majority
I20251215 21:06:03.788923 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 84 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 83. Current term is 84. Ops: []
I20251215 21:06:03.789297 24072 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 83.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251215 21:06:03.789464 24072 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 LEADER]: Stepping down as leader of term 83
I20251215 21:06:03.789510 24072 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:03.789641 24072 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 83.81, Last appended by leader: 81, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.789784 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 83 FOLLOWER]: Advancing to term 84
I20251215 21:06:03.869973 24073 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 83 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:03.870098 24073 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 83 FOLLOWER]: Advancing to term 84
I20251215 21:06:03.870939 24073 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.871312 24073 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 84 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:03.871735 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 84 candidate_status { last_received { term: 83 index: 81 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:03.871739 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 84 candidate_status { last_received { term: 83 index: 81 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:03.871923 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 84 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 84: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:03.872573 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 84.
I20251215 21:06:03.872852 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:03.873024 24073 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 84 FOLLOWER]: Leader election won for term 84
I20251215 21:06:03.873107 24073 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 84 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:03.873247 24073 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 83.81, Last appended by leader: 81, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.874521 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 83 to 84, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 84 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:03.903512 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 84 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:03.903614 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 84 FOLLOWER]: Advancing to term 85
I20251215 21:06:03.904412 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.904759 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 85 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:03.905145 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 85 candidate_status { last_received { term: 83 index: 81 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:03.905146 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 85 candidate_status { last_received { term: 83 index: 81 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:03.905285 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 84 FOLLOWER]: Advancing to term 85
I20251215 21:06:03.905995 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 85.
I20251215 21:06:03.906250 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:03.906392 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 FOLLOWER]: Leader election won for term 85
I20251215 21:06:03.906477 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:03.906625 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 83.81, Last appended by leader: 81, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.907812 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 84 to 85, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 85 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:03.956772 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 84. Current term is 85. Ops: []
I20251215 21:06:03.957230 24073 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251215 21:06:03.957373 24073 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 85.82 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 81
I20251215 21:06:03.957430 24075 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 84 LEADER]: Stepping down as leader of term 84
I20251215 21:06:03.957491 24075 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:03.957628 24075 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 84.82, Last appended by leader: 82, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:03.957767 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 84 FOLLOWER]: Advancing to term 85
I20251215 21:06:03.999815 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 85 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 84 index: 82. Preceding OpId from leader: term: 85 index: 82. (term mismatch)
I20251215 21:06:03.999903 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 81
I20251215 21:06:03.999928 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 84.82
I20251215 21:06:03.999953 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 85 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:04.000349 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251215 21:06:04.002395 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 85 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 83 index: 81. Preceding OpId from leader: term: 85 index: 82. (index mismatch)
I20251215 21:06:04.002787 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251215 21:06:04.216554 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 85 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:04.216699 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 85 FOLLOWER]: Advancing to term 86
I20251215 21:06:04.217648 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.218061 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 86 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:04.218477 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 86 candidate_status { last_received { term: 85 index: 82 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:04.218474 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 86 candidate_status { last_received { term: 85 index: 82 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:04.218842 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 86 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:04.218999 24075 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 86 FOLLOWER]: Leader election lost for term 86. Reason: could not achieve majority
I20251215 21:06:04.223328 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 86 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 85. Current term is 86. Ops: []
I20251215 21:06:04.223650 24076 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 85.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251215 21:06:04.223816 24056 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 LEADER]: Stepping down as leader of term 85
I20251215 21:06:04.223870 24056 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:04.223977 24056 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 85.82, Last appended by leader: 82, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.224102 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 85 FOLLOWER]: Advancing to term 86
I20251215 21:06:04.277926 24078 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 85 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:04.278066 24078 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 85 FOLLOWER]: Advancing to term 86
I20251215 21:06:04.279006 24078 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.279413 24078 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 86 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:04.279816 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 86 candidate_status { last_received { term: 85 index: 82 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:04.279816 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 86 candidate_status { last_received { term: 85 index: 82 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:04.279987 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 86 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 86: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:04.280681 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 86.
I20251215 21:06:04.280968 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:04.281172 24078 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 86 FOLLOWER]: Leader election won for term 86
I20251215 21:06:04.281275 24078 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 86 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:04.281435 24078 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 85.82, Last appended by leader: 82, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.282778 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 85 to 86, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 86 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:04.291209 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:04.348527 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 86 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:04.348659 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 86 FOLLOWER]: Advancing to term 87
I20251215 21:06:04.349603 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.350023 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 87 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:04.350416 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 87 candidate_status { last_received { term: 85 index: 82 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:04.350446 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 87 candidate_status { last_received { term: 85 index: 82 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:04.350590 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 86 FOLLOWER]: Advancing to term 87
I20251215 21:06:04.351390 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 87.
I20251215 21:06:04.351645 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:04.351804 24075 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 FOLLOWER]: Leader election won for term 87
I20251215 21:06:04.351888 24075 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:04.352039 24075 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 85.82, Last appended by leader: 82, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.353310 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 86 to 87, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 87 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:04.373004 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 86. Current term is 87. Ops: []
I20251215 21:06:04.373004 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 87 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 86. Current term is 87. Ops: []
I20251215 21:06:04.373517 24078 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251215 21:06:04.373770 24078 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 87.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 82
I20251215 21:06:04.373843 24080 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 86 LEADER]: Stepping down as leader of term 86
I20251215 21:06:04.373926 24080 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 86 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:04.373855 24072 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251215 21:06:04.374177 24080 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 86.83, Last appended by leader: 83, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.374336 24080 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 86 FOLLOWER]: Advancing to term 87
I20251215 21:06:04.433679 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 87 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 85 index: 82. Preceding OpId from leader: term: 87 index: 83. (index mismatch)
I20251215 21:06:04.434096 24075 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251215 21:06:04.436228 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 87 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 86 index: 83. Preceding OpId from leader: term: 87 index: 83. (term mismatch)
I20251215 21:06:04.436295 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 82
I20251215 21:06:04.436317 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 86.83
I20251215 21:06:04.436340 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 87 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:04.436661 24082 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251215 21:06:04.536979 24056 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 87 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:04.537118 24056 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 87 FOLLOWER]: Advancing to term 88
I20251215 21:06:04.538009 24056 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.538419 24056 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 88 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:04.538830 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 88 candidate_status { last_received { term: 87 index: 83 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:04.538833 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 88 candidate_status { last_received { term: 87 index: 83 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:04.538975 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 87 FOLLOWER]: Advancing to term 88
I20251215 21:06:04.539723 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 88.
I20251215 21:06:04.539983 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:04.540133 24056 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 88 FOLLOWER]: Leader election won for term 88
I20251215 21:06:04.540230 24056 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 88 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:04.540364 24056 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 87.83, Last appended by leader: 83, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.541638 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 87 to 88, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 88 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:04.550124 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 88 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 87. Current term is 88. Ops: []
I20251215 21:06:04.550546 24081 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 87.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20251215 21:06:04.550727 24075 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 LEADER]: Stepping down as leader of term 87
I20251215 21:06:04.550787 24075 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:04.550918 24075 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 87.83, Last appended by leader: 83, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.551100 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 87 FOLLOWER]: Advancing to term 88
I20251215 21:06:04.634896 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 88 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 87 index: 83. Preceding OpId from leader: term: 88 index: 84. (index mismatch)
I20251215 21:06:04.635321 24056 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20251215 21:06:04.637427 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 88 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 87 index: 83. Preceding OpId from leader: term: 88 index: 84. (index mismatch)
I20251215 21:06:04.637868 24083 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20251215 21:06:04.739125 24080 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 88 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:04.739274 24080 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 88 FOLLOWER]: Advancing to term 89
I20251215 21:06:04.740257 24080 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.740703 24080 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 89 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:04.741128 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 89 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:04.741128 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 89 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:04.741559 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 89 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:04.741750 24080 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 89 FOLLOWER]: Leader election lost for term 89. Reason: could not achieve majority
I20251215 21:06:04.752199 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 89 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 88. Current term is 89. Ops: []
I20251215 21:06:04.752560 24083 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 88.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251215 21:06:04.752743 24076 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 88 LEADER]: Stepping down as leader of term 88
I20251215 21:06:04.752832 24076 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 88 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:04.752972 24076 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 88.84, Last appended by leader: 84, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.753141 24076 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 88 FOLLOWER]: Advancing to term 89
I20251215 21:06:04.823832 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 88 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:04.823957 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 88 FOLLOWER]: Advancing to term 89
I20251215 21:06:04.824887 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.825291 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 89 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:04.825703 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 89 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:04.825719 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 89 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:04.825867 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 89: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:04.826576 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 89.
I20251215 21:06:04.826818 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:04.826956 24075 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 89 FOLLOWER]: Leader election won for term 89
I20251215 21:06:04.827045 24075 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 89 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:04.827183 24075 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 88.84, Last appended by leader: 84, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.828514 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 88 to 89, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 89 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:04.861882 24080 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 89 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:04.862028 24080 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 89 FOLLOWER]: Advancing to term 90
I20251215 21:06:04.863157 24080 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.863550 24080 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 90 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:04.863993 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 90 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:04.863993 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 90 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:04.864143 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 89 FOLLOWER]: Advancing to term 90
I20251215 21:06:04.864970 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 90.
I20251215 21:06:04.865239 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:04.865398 24080 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 FOLLOWER]: Leader election won for term 90
I20251215 21:06:04.865489 24080 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:04.865629 24080 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 88.84, Last appended by leader: 84, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.866895 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 89 to 90, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 90 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:04.928781 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 89. Current term is 90. Ops: []
I20251215 21:06:04.929250 24075 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251215 21:06:04.929410 24075 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 90.85 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 84
I20251215 21:06:04.929459 24081 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 89 LEADER]: Stepping down as leader of term 89
I20251215 21:06:04.929514 24081 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 89 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:04.929565 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 90 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 89. Current term is 90. Ops: []
I20251215 21:06:04.929649 24081 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 89.85, Last appended by leader: 85, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.929826 24081 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 89 FOLLOWER]: Advancing to term 90
I20251215 21:06:04.978328 24076 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 90 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:04.978464 24076 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 90 FOLLOWER]: Advancing to term 91
I20251215 21:06:04.979159 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 90 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 89 index: 85. Preceding OpId from leader: term: 90 index: 85. (term mismatch)
I20251215 21:06:04.979236 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 84
I20251215 21:06:04.979261 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 89.85
I20251215 21:06:04.979290 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 90 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:04.979632 24076 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.979672 24080 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251215 21:06:04.980026 24076 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 91 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:04.980295 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 91 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:04.980329 23901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 91 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:04.980693 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 91 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:04.980887 24076 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 91 FOLLOWER]: Leader election lost for term 91. Reason: could not achieve majority
I20251215 21:06:04.981436 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 91 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 90. Current term is 91. Ops: []
I20251215 21:06:04.981822 24084 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251215 21:06:04.981966 24072 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 LEADER]: Stepping down as leader of term 90
I20251215 21:06:04.982021 24072 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:04.982142 24072 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 90.85, Last appended by leader: 85, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:04.982288 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 90 FOLLOWER]: Advancing to term 91
I20251215 21:06:05.093396 24076 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 91 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:05.093523 24076 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 91 FOLLOWER]: Advancing to term 92
I20251215 21:06:05.094413 24076 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.094797 24076 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 92 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:05.095211 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 92 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:05.095211 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 92 candidate_status { last_received { term: 88 index: 84 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.095351 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 91 FOLLOWER]: Advancing to term 92
I20251215 21:06:05.095351 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 90 FOLLOWER]: Advancing to term 92
I20251215 21:06:05.096267 23902 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b for term 92 because replica has last-logged OpId of term: 90 index: 85, which is greater than that of the candidate, which has last-logged OpId of term: 88 index: 84.
I20251215 21:06:05.096267 23751 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b for term 92 because replica has last-logged OpId of term: 90 index: 85, which is greater than that of the candidate, which has last-logged OpId of term: 88 index: 84.
I20251215 21:06:05.096612 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:05.096774 24076 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I20251215 21:06:05.154899 24081 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 92 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:05.155050 24081 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 92 FOLLOWER]: Advancing to term 93
I20251215 21:06:05.155980 24081 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.156380 24081 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 93 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:05.156862 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 93 candidate_status { last_received { term: 90 index: 85 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:05.156862 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 93 candidate_status { last_received { term: 90 index: 85 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:05.157011 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 92 FOLLOWER]: Advancing to term 93
I20251215 21:06:05.157011 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 92 FOLLOWER]: Advancing to term 93
I20251215 21:06:05.157945 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 93.
I20251215 21:06:05.157954 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 93.
I20251215 21:06:05.158278 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:06:05.158440 24081 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 93 FOLLOWER]: Leader election won for term 93
I20251215 21:06:05.158521 24081 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 93 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:05.158663 24081 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 90.85, Last appended by leader: 85, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.159938 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 90 to 93, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 93 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:05.242187 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 93 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 88 index: 84. Preceding OpId from leader: term: 93 index: 86. (index mismatch)
I20251215 21:06:05.242606 24081 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20251215 21:06:05.244663 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 93 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 90 index: 85. Preceding OpId from leader: term: 93 index: 86. (index mismatch)
I20251215 21:06:05.245059 24081 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20251215 21:06:05.291440 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:05.345515 24076 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 93 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:05.345640 24076 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 93 FOLLOWER]: Advancing to term 94
I20251215 21:06:05.346096 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 93 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:05.346175 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 93 FOLLOWER]: Advancing to term 94
I20251215 21:06:05.346722 24076 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.347062 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.347363 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 94 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:05.347347 24076 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 94 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:05.347606 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 94 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:05.347611 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 94 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.347739 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 94: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:05.347739 23901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 94 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.347611 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 94 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:05.348049 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 94: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:05.348143 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:05.348297 24076 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20251215 21:06:05.348448 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:05.348619 24072 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20251215 21:06:05.357779 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 94 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 93. Current term is 94. Ops: []
I20251215 21:06:05.358124 24081 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 93.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251215 21:06:05.358290 24081 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 93 LEADER]: Stepping down as leader of term 93
I20251215 21:06:05.358332 24081 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:05.358461 24081 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 93.86, Last appended by leader: 86, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.358601 24081 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 93 FOLLOWER]: Advancing to term 94
I20251215 21:06:05.482852 24076 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 94 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:05.482990 24076 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 94 FOLLOWER]: Advancing to term 95
I20251215 21:06:05.483827 24076 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.484216 24076 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 95 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:05.484612 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 95 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:05.484612 23901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 95 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.484730 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 94 FOLLOWER]: Advancing to term 95
I20251215 21:06:05.484738 23901 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 94 FOLLOWER]: Advancing to term 95
I20251215 21:06:05.485517 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 95.
I20251215 21:06:05.485587 23901 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 95.
I20251215 21:06:05.485805 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:06:05.486008 24076 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 95 FOLLOWER]: Leader election won for term 95
I20251215 21:06:05.486142 24076 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 95 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:05.486284 24076 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 93.86, Last appended by leader: 86, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.487563 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 93 to 95, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 95 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:05.590333 24072 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 95 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:05.590461 24072 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 95 FOLLOWER]: Advancing to term 96
I20251215 21:06:05.591830 24072 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.592231 24072 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 96 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:05.592563 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 96 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:05.592579 23901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 96 candidate_status { last_received { term: 93 index: 86 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.592650 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 95 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 93 index: 86. Preceding OpId from leader: term: 95 index: 87. (index mismatch)
I20251215 21:06:05.592931 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:05.592996 24076 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251215 21:06:05.593081 24072 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
I20251215 21:06:05.594733 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 95. Current term is 96. Ops: []
I20251215 21:06:05.595086 24085 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251215 21:06:05.595323 24085 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 95 LEADER]: Stepping down as leader of term 95
I20251215 21:06:05.595430 24085 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 95 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:05.595595 24085 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 95.87, Last appended by leader: 87, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.595741 24085 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 95 FOLLOWER]: Advancing to term 96
W20251215 21:06:05.595760 24076 consensus_queue.cc:1175] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "31ff5cfb5d624764ae78ac08450e9022" responder_term: 95 status { last_received { term: 95 index: 87 } last_committed_idx: 87 last_received_current_leader { term: 95 index: 87 } }
I20251215 21:06:05.695834 24081 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 95 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:05.695993 24081 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 95 FOLLOWER]: Advancing to term 96
I20251215 21:06:05.696899 24081 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.697305 24081 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 96 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:05.697759 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 96 candidate_status { last_received { term: 95 index: 87 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:05.697759 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 96 candidate_status { last_received { term: 95 index: 87 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:05.697937 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 96: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:05.698734 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 96.
I20251215 21:06:05.698974 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:05.699131 24081 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 96 FOLLOWER]: Leader election won for term 96
I20251215 21:06:05.699214 24081 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 96 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:05.699384 24081 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 95.87, Last appended by leader: 87, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.700692 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 95 to 96, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 96 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:05.811872 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 96 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 95 index: 87. Preceding OpId from leader: term: 96 index: 88. (index mismatch)
I20251215 21:06:05.812273 24081 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20251215 21:06:05.814129 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 93 index: 86. Preceding OpId from leader: term: 96 index: 88. (index mismatch)
I20251215 21:06:05.814579 24087 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20251215 21:06:05.914522 24085 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 96 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:05.914659 24085 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 96 FOLLOWER]: Advancing to term 97
I20251215 21:06:05.915560 24085 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.915722 24088 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:05.915827 24088 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 96 FOLLOWER]: Advancing to term 97
I20251215 21:06:05.915994 24085 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 97 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:05.916348 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 97 candidate_status { last_received { term: 96 index: 88 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.916349 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 97 candidate_status { last_received { term: 96 index: 88 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:05.916855 24088 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.917052 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 97 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 97: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:05.917207 24088 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 97 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:05.917459 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:05.917479 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 97 candidate_status { last_received { term: 96 index: 88 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:05.917479 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 97 candidate_status { last_received { term: 96 index: 88 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:05.917611 24085 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20251215 21:06:05.917766 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 97 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 97: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:05.917991 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:05.918128 24088 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20251215 21:06:05.920948 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 97 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 96. Current term is 97. Ops: []
I20251215 21:06:05.921247 24081 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 96.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251215 21:06:05.921398 24075 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 96 LEADER]: Stepping down as leader of term 96
I20251215 21:06:05.921447 24075 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 96 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:05.921569 24075 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 96.88, Last appended by leader: 88, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:05.921697 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 96 FOLLOWER]: Advancing to term 97
I20251215 21:06:06.020578 24088 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 97 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:06.020721 24088 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 97 FOLLOWER]: Advancing to term 98
I20251215 21:06:06.021693 24088 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.022114 24088 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 98 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:06.022533 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 98 candidate_status { last_received { term: 96 index: 88 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:06.022533 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 98 candidate_status { last_received { term: 96 index: 88 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:06.022680 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 97 FOLLOWER]: Advancing to term 98
I20251215 21:06:06.022680 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 97 FOLLOWER]: Advancing to term 98
I20251215 21:06:06.023624 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 98.
I20251215 21:06:06.023624 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 98.
I20251215 21:06:06.023926 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters:
I20251215 21:06:06.024112 24088 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 98 FOLLOWER]: Leader election won for term 98
I20251215 21:06:06.024211 24088 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 98 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:06.024362 24088 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 96.88, Last appended by leader: 88, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.025751 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 96 to 98, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 98 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:06.117827 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 98 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 96 index: 88. Preceding OpId from leader: term: 98 index: 89. (index mismatch)
I20251215 21:06:06.118199 24088 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251215 21:06:06.120033 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 98 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 96 index: 88. Preceding OpId from leader: term: 98 index: 89. (index mismatch)
I20251215 21:06:06.120348 24088 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251215 21:06:06.221302 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 98 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:06.221431 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 98 FOLLOWER]: Advancing to term 99
I20251215 21:06:06.222312 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.222700 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 99 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:06.223058 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 99 candidate_status { last_received { term: 98 index: 89 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:06.223091 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 99 candidate_status { last_received { term: 98 index: 89 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:06.223425 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:06.223572 24075 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20251215 21:06:06.234292 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 99 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 98. Current term is 99. Ops: []
I20251215 21:06:06.234644 24088 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 98.89, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20251215 21:06:06.234826 24089 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 98 LEADER]: Stepping down as leader of term 98
I20251215 21:06:06.234879 24089 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 98 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:06.235005 24089 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 98.89, Last appended by leader: 89, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.235133 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 98 FOLLOWER]: Advancing to term 99
I20251215 21:06:06.291736 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:06.307724 24085 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 98 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:06.307848 24085 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 98 FOLLOWER]: Advancing to term 99
I20251215 21:06:06.308712 24085 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.309105 24085 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 99 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:06.309514 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 99 candidate_status { last_received { term: 98 index: 89 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:06.309520 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 99 candidate_status { last_received { term: 98 index: 89 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:06.309651 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 99: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:06.310374 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 99 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 99.
I20251215 21:06:06.310616 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 99 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:06.310778 24085 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 99 FOLLOWER]: Leader election won for term 99
I20251215 21:06:06.310918 24085 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 99 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:06.311054 24085 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 98.89, Last appended by leader: 89, Current term: 99, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.312286 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 98 to 99, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 99 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:06.363015 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:06.363142 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 99 FOLLOWER]: Advancing to term 100
I20251215 21:06:06.364006 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.364379 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 100 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:06.364790 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 100 candidate_status { last_received { term: 98 index: 89 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:06.364790 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 100 candidate_status { last_received { term: 98 index: 89 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:06.364952 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 99 FOLLOWER]: Advancing to term 100
I20251215 21:06:06.365662 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 100.
I20251215 21:06:06.365911 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:06.366257 24075 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 100 FOLLOWER]: Leader election won for term 100
I20251215 21:06:06.366366 24075 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 100 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:06.366503 24075 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 98.89, Last appended by leader: 89, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.367853 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 99 to 100, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 100 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:06.387398 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 100 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 99. Current term is 100. Ops: []
I20251215 21:06:06.387784 24085 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20251215 21:06:06.387977 24090 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 99 LEADER]: Stepping down as leader of term 99
I20251215 21:06:06.388031 24090 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 99 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:06.388159 24090 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 99.90, Last appended by leader: 90, Current term: 99, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.388298 24090 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 99 FOLLOWER]: Advancing to term 100
I20251215 21:06:06.473100 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 100 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 98 index: 89. Preceding OpId from leader: term: 100 index: 90. (index mismatch)
I20251215 21:06:06.473533 24075 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20251215 21:06:06.475847 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 100 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 99 index: 90. Preceding OpId from leader: term: 100 index: 90. (term mismatch)
I20251215 21:06:06.475942 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 89
I20251215 21:06:06.475984 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 99.90
I20251215 21:06:06.476018 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 100 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:06.476742 24091 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20251215 21:06:06.676237 24089 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 100 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:06.676373 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 100 FOLLOWER]: Advancing to term 101
I20251215 21:06:06.677318 24089 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.677711 24089 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 101 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:06.678117 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 101 candidate_status { last_received { term: 100 index: 90 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:06.678117 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 101 candidate_status { last_received { term: 100 index: 90 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:06.678529 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:06.678694 24089 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20251215 21:06:06.698880 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 100. Current term is 101. Ops: []
I20251215 21:06:06.699301 24075 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 100.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20251215 21:06:06.699492 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 100 LEADER]: Stepping down as leader of term 100
I20251215 21:06:06.699568 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:06.699718 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 100.90, Last appended by leader: 90, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.699898 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 100 FOLLOWER]: Advancing to term 101
I20251215 21:06:06.701591 24090 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 100 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:06.701659 24090 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 100 FOLLOWER]: Advancing to term 101
I20251215 21:06:06.702369 24090 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.702695 24090 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 101 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:06.703025 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 101 candidate_status { last_received { term: 100 index: 90 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:06.703032 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 101 candidate_status { last_received { term: 100 index: 90 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:06.703225 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 101: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:06.703745 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 101.
I20251215 21:06:06.703981 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 101 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:06.704123 24090 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 101 FOLLOWER]: Leader election won for term 101
I20251215 21:06:06.704203 24090 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 101 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:06.704339 24090 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 100.90, Last appended by leader: 90, Current term: 101, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.705610 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 100 to 101, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 101 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:06.786911 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 100 index: 90. Preceding OpId from leader: term: 101 index: 91. (index mismatch)
I20251215 21:06:06.787325 24090 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20251215 21:06:06.789810 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 101 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 100 index: 90. Preceding OpId from leader: term: 101 index: 91. (index mismatch)
I20251215 21:06:06.790354 24090 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20251215 21:06:06.966464 24089 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:06.966579 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 101 FOLLOWER]: Advancing to term 102
I20251215 21:06:06.967455 24089 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.967666 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 102 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 101. Current term is 102. Ops: []
I20251215 21:06:06.967876 24089 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 102 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:06.968169 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 102 candidate_status { last_received { term: 101 index: 91 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:06.968179 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 102 candidate_status { last_received { term: 101 index: 91 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:06.968240 24092 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 101.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251215 21:06:06.968448 24090 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 101 LEADER]: Stepping down as leader of term 101
I20251215 21:06:06.968545 24090 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 101 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:06.968546 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 102 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:06.968786 24089 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 102 FOLLOWER]: Leader election lost for term 102. Reason: could not achieve majority
I20251215 21:06:06.968742 24090 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 101.91, Last appended by leader: 91, Current term: 101, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:06.968950 24090 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 101 FOLLOWER]: Advancing to term 102
I20251215 21:06:07.016077 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 101 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:07.016222 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 101 FOLLOWER]: Advancing to term 102
I20251215 21:06:07.017151 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.017534 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 102 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:07.017931 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 102 candidate_status { last_received { term: 101 index: 91 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:07.017962 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 102 candidate_status { last_received { term: 101 index: 91 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:07.018128 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 102: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:07.018707 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 102.
I20251215 21:06:07.018949 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:07.019104 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 102 FOLLOWER]: Leader election won for term 102
I20251215 21:06:07.019188 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 102 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:07.019325 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 101.91, Last appended by leader: 91, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.020592 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 101 to 102, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 102 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:07.089280 24089 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 102 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:07.089416 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 102 FOLLOWER]: Advancing to term 103
I20251215 21:06:07.090314 24089 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.090703 24089 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 103 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:07.091130 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 103 candidate_status { last_received { term: 101 index: 91 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:07.091179 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 103 candidate_status { last_received { term: 101 index: 91 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:07.091241 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 102 FOLLOWER]: Advancing to term 103
I20251215 21:06:07.091955 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 103.
I20251215 21:06:07.092257 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 103 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:07.092439 24089 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 103 FOLLOWER]: Leader election won for term 103
I20251215 21:06:07.092604 24089 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 103 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:07.092749 24089 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 101.91, Last appended by leader: 91, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.094086 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 102 to 103, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 103 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:07.095206 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 103 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 102. Current term is 103. Ops: []
I20251215 21:06:07.095613 24091 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251215 21:06:07.095819 24075 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 102 LEADER]: Stepping down as leader of term 102
I20251215 21:06:07.095881 24075 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 102 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:07.096004 24075 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 102.92, Last appended by leader: 92, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.096144 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 102 FOLLOWER]: Advancing to term 103
I20251215 21:06:07.203303 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 103 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 101 index: 91. Preceding OpId from leader: term: 103 index: 92. (index mismatch)
I20251215 21:06:07.203706 24089 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251215 21:06:07.205735 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 103 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 102 index: 92. Preceding OpId from leader: term: 103 index: 92. (term mismatch)
I20251215 21:06:07.205798 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 91
I20251215 21:06:07.205821 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 102.92
I20251215 21:06:07.205843 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 103 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:07.206286 24093 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251215 21:06:07.292016 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:07.405925 24090 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 103 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:07.406049 24090 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 103 FOLLOWER]: Advancing to term 104
I20251215 21:06:07.406859 24090 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.407220 24090 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 104 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:07.407560 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 104 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:07.407567 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 104 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:07.407919 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 104 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:07.408079 24090 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 104 FOLLOWER]: Leader election lost for term 104. Reason: could not achieve majority
I20251215 21:06:07.422719 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 104 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 103. Current term is 104. Ops: []
I20251215 21:06:07.423094 24094 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 103.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20251215 21:06:07.423307 24089 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 103 LEADER]: Stepping down as leader of term 103
I20251215 21:06:07.423388 24089 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 103 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:07.423677 24089 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 103.92, Last appended by leader: 92, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.423874 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 103 FOLLOWER]: Advancing to term 104
I20251215 21:06:07.496021 24075 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 103 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:07.496168 24075 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 103 FOLLOWER]: Advancing to term 104
I20251215 21:06:07.497097 24075 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.497490 24075 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 104 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:07.497915 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 104 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:07.497915 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 104 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:07.498093 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 104 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 104: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:07.498852 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 104.
I20251215 21:06:07.499104 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:07.499259 24075 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 104 FOLLOWER]: Leader election won for term 104
I20251215 21:06:07.499341 24075 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 104 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:07.499475 24075 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 103.92, Last appended by leader: 92, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.500747 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 103 to 104, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 104 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:07.557690 24090 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 104 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:07.557819 24090 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 104 FOLLOWER]: Advancing to term 105
I20251215 21:06:07.558693 24090 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.559069 24090 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 105 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:07.559490 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 105 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:07.559487 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 105 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:07.559603 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 104 FOLLOWER]: Advancing to term 105
I20251215 21:06:07.560360 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 105 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 105.
I20251215 21:06:07.560609 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 105 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:07.560786 24090 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 FOLLOWER]: Leader election won for term 105
I20251215 21:06:07.560977 24090 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:07.561122 24090 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 103.92, Last appended by leader: 92, Current term: 105, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.562377 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 104 to 105, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 105 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:07.616981 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 104. Current term is 105. Ops: []
I20251215 21:06:07.617374 24075 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20251215 21:06:07.617520 24075 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 105.93 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 92
I20251215 21:06:07.617578 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 104 LEADER]: Stepping down as leader of term 104
I20251215 21:06:07.617642 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:07.617779 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 104.93, Last appended by leader: 93, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.617920 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 104 FOLLOWER]: Advancing to term 105
I20251215 21:06:07.667348 24089 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:07.667483 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 105 FOLLOWER]: Advancing to term 106
I20251215 21:06:07.668380 24089 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.668772 24089 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 106 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:07.669221 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 106 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:07.669221 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 106 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:07.669369 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 105 FOLLOWER]: Advancing to term 106
I20251215 21:06:07.670113 23902 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for term 106 because replica has last-logged OpId of term: 104 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 103 index: 92.
I20251215 21:06:07.670356 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 106 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:07.670493 24089 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 106 FOLLOWER]: Leader election lost for term 106. Reason: could not achieve majority
I20251215 21:06:07.679425 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 106 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 105. Current term is 106. Ops: []
I20251215 21:06:07.679839 24090 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20251215 21:06:07.680022 24095 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 LEADER]: Stepping down as leader of term 105
I20251215 21:06:07.680090 24095 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:07.680227 24095 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 105.93, Last appended by leader: 93, Current term: 105, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.680368 24095 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 105 FOLLOWER]: Advancing to term 106
I20251215 21:06:07.773000 24089 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 106 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:07.773132 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 106 FOLLOWER]: Advancing to term 107
I20251215 21:06:07.774029 24089 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.774432 24089 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 107 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:07.774842 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 107 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:07.774850 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 107 candidate_status { last_received { term: 103 index: 92 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:07.774950 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 106 FOLLOWER]: Advancing to term 107
I20251215 21:06:07.774976 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 106 FOLLOWER]: Advancing to term 107
I20251215 21:06:07.775701 23902 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for term 107 because replica has last-logged OpId of term: 104 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 103 index: 92.
I20251215 21:06:07.775708 23826 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef for term 107 because replica has last-logged OpId of term: 105 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 103 index: 92.
I20251215 21:06:07.776029 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 107 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:07.776176 24089 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 107 FOLLOWER]: Leader election lost for term 107. Reason: could not achieve majority
I20251215 21:06:07.882452 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:07.882576 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 107 FOLLOWER]: Advancing to term 108
I20251215 21:06:07.883484 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.883889 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 108 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:07.884300 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 108 candidate_status { last_received { term: 104 index: 93 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:07.884300 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 108 candidate_status { last_received { term: 104 index: 93 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:07.884449 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 107 FOLLOWER]: Advancing to term 108
I20251215 21:06:07.884449 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 107 FOLLOWER]: Advancing to term 108
I20251215 21:06:07.885390 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 108.
I20251215 21:06:07.885438 23826 raft_consensus.cc:2410] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 for term 108 because replica has last-logged OpId of term: 105 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 104 index: 93.
I20251215 21:06:07.885690 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters:
I20251215 21:06:07.885847 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 108 FOLLOWER]: Leader election won for term 108
I20251215 21:06:07.885953 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 108 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:07.886098 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 104.93, Last appended by leader: 93, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:07.887414 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 105 to 108, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 108 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:07.986739 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 108 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 105 index: 93. Preceding OpId from leader: term: 108 index: 94. (index mismatch)
I20251215 21:06:07.987160 24091 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20251215 21:06:07.987293 24091 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 105.93 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 92
I20251215 21:06:07.987775 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 92
I20251215 21:06:07.987823 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 105.93
I20251215 21:06:07.987850 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:07.989477 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 108 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 103 index: 92. Preceding OpId from leader: term: 108 index: 94. (index mismatch)
I20251215 21:06:07.989797 24096 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20251215 21:06:08.089862 24095 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 108 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:08.090008 24095 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 108 FOLLOWER]: Advancing to term 109
I20251215 21:06:08.090906 24095 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.091315 24095 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 109 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:08.091692 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 109 candidate_status { last_received { term: 108 index: 94 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:08.091699 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 109 candidate_status { last_received { term: 108 index: 94 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:08.092084 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 109 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:08.092265 24095 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 109 FOLLOWER]: Leader election lost for term 109. Reason: could not achieve majority
I20251215 21:06:08.092840 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 109 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 108. Current term is 109. Ops: []
I20251215 21:06:08.093152 24096 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 108.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251215 21:06:08.093333 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 108 LEADER]: Stepping down as leader of term 108
I20251215 21:06:08.093390 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 108 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:08.093518 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 108.94, Last appended by leader: 94, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.093658 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 108 FOLLOWER]: Advancing to term 109
I20251215 21:06:08.173313 24097 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 108 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:08.173440 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 108 FOLLOWER]: Advancing to term 109
I20251215 21:06:08.174346 24097 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.174741 24097 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 109 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:08.175108 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 109 candidate_status { last_received { term: 108 index: 94 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:08.175125 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 109 candidate_status { last_received { term: 108 index: 94 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:08.175230 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 109 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 109: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:08.175967 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 109.
I20251215 21:06:08.176208 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:08.176367 24097 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 109 FOLLOWER]: Leader election won for term 109
I20251215 21:06:08.176448 24097 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 109 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:08.176582 24097 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 108.94, Last appended by leader: 94, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.177911 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 108 to 109, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 109 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:08.205402 24095 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 109 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:08.205518 24095 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 109 FOLLOWER]: Advancing to term 110
I20251215 21:06:08.206382 24095 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.206740 24095 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 110 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:08.207103 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 110 candidate_status { last_received { term: 108 index: 94 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:08.207108 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 110 candidate_status { last_received { term: 108 index: 94 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:08.207227 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 109 FOLLOWER]: Advancing to term 110
I20251215 21:06:08.207970 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 110.
I20251215 21:06:08.208212 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 110 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:08.208359 24095 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 FOLLOWER]: Leader election won for term 110
I20251215 21:06:08.208524 24095 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:08.208657 24095 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 108.94, Last appended by leader: 94, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.209947 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 109 to 110, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 110 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:08.256498 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 LEADER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 109. Current term is 110. Ops: []
I20251215 21:06:08.256953 24097 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251215 21:06:08.257102 24097 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 110.95 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 94
I20251215 21:06:08.257157 24089 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 109 LEADER]: Stepping down as leader of term 109
I20251215 21:06:08.257230 24089 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 109 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:08.257365 24089 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 109.95, Last appended by leader: 95, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.257501 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 109 FOLLOWER]: Advancing to term 110
I20251215 21:06:08.292306 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:08.305352 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 110 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 109 index: 95. Preceding OpId from leader: term: 110 index: 95. (term mismatch)
I20251215 21:06:08.305441 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 94
I20251215 21:06:08.305459 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 109.95
I20251215 21:06:08.305476 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 110 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:08.305826 24095 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251215 21:06:08.308075 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 110 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 108 index: 94. Preceding OpId from leader: term: 110 index: 95. (index mismatch)
I20251215 21:06:08.308449 24095 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251215 21:06:08.578405 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 110 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:08.578533 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 110 FOLLOWER]: Advancing to term 111
I20251215 21:06:08.579408 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.579797 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 111 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:08.580192 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 111 candidate_status { last_received { term: 110 index: 95 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:08.580193 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 111 candidate_status { last_received { term: 110 index: 95 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:08.580543 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 111 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:08.580694 24091 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Leader election lost for term 111. Reason: could not achieve majority
I20251215 21:06:08.582088 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 110. Current term is 111. Ops: []
I20251215 21:06:08.582409 24095 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 110.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20251215 21:06:08.582561 24101 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 LEADER]: Stepping down as leader of term 110
I20251215 21:06:08.582612 24101 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:08.582723 24101 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 110.95, Last appended by leader: 95, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.582867 24101 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 110 FOLLOWER]: Advancing to term 111
I20251215 21:06:08.608305 24089 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 110 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:08.608439 24089 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 110 FOLLOWER]: Advancing to term 111
I20251215 21:06:08.609352 24089 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.609733 24089 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 111 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:08.610217 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 111 candidate_status { last_received { term: 110 index: 95 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:08.610252 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 111 candidate_status { last_received { term: 110 index: 95 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:08.610340 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 111: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:08.611061 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 111.
I20251215 21:06:08.611335 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:08.611495 24089 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 111 FOLLOWER]: Leader election won for term 111
I20251215 21:06:08.611580 24089 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 111 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:08.611716 24089 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 110.95, Last appended by leader: 95, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.613042 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 110 to 111, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 111 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:08.690268 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 110 index: 95. Preceding OpId from leader: term: 111 index: 96. (index mismatch)
I20251215 21:06:08.690677 24089 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20251215 21:06:08.692580 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 111 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 110 index: 95. Preceding OpId from leader: term: 111 index: 96. (index mismatch)
I20251215 21:06:08.693041 24089 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20251215 21:06:08.892699 24095 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 111 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:08.892843 24095 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 111 FOLLOWER]: Advancing to term 112
I20251215 21:06:08.893713 24095 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.894102 24095 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 112 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:08.894487 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 112 candidate_status { last_received { term: 111 index: 96 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:08.894491 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 112 candidate_status { last_received { term: 111 index: 96 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:08.894866 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:08.895026 24095 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20251215 21:06:08.905745 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 112 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 111. Current term is 112. Ops: []
I20251215 21:06:08.906140 24102 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 111.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251215 21:06:08.906340 24097 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 111 LEADER]: Stepping down as leader of term 111
I20251215 21:06:08.906416 24097 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 111 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:08.906586 24097 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 111.96, Last appended by leader: 96, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.906749 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 111 FOLLOWER]: Advancing to term 112
I20251215 21:06:08.984961 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:08.985102 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 111 FOLLOWER]: Advancing to term 112
I20251215 21:06:08.985986 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.986380 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 112 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:08.986776 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 112 candidate_status { last_received { term: 111 index: 96 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:08.986776 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 112 candidate_status { last_received { term: 111 index: 96 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:08.986951 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 112: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:08.987699 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 112.
I20251215 21:06:08.987965 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 112 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:08.988116 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 112 FOLLOWER]: Leader election won for term 112
I20251215 21:06:08.988272 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 112 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:08.988405 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 111.96, Last appended by leader: 96, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:08.989720 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 111 to 112, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 112 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:09.040529 24095 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 112 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:09.040663 24095 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 112 FOLLOWER]: Advancing to term 113
I20251215 21:06:09.041579 24095 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.041994 24095 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 113 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:09.042409 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 113 candidate_status { last_received { term: 111 index: 96 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:09.042409 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 113 candidate_status { last_received { term: 111 index: 96 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:09.042560 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 112 FOLLOWER]: Advancing to term 113
I20251215 21:06:09.043305 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 113.
I20251215 21:06:09.043540 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:09.043681 24095 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 FOLLOWER]: Leader election won for term 113
I20251215 21:06:09.043772 24095 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:09.043910 24095 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 111.96, Last appended by leader: 96, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.045233 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 112 to 113, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 113 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:09.090778 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 112. Current term is 113. Ops: []
I20251215 21:06:09.091152 24091 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251215 21:06:09.091290 24091 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 113.97 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 96
I20251215 21:06:09.091353 24103 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 112 LEADER]: Stepping down as leader of term 112
I20251215 21:06:09.091425 24103 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 112 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:09.091569 24103 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 112.97, Last appended by leader: 97, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.091737 24103 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 112 FOLLOWER]: Advancing to term 113
I20251215 21:06:09.130384 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 113 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 112 index: 97. Preceding OpId from leader: term: 113 index: 97. (term mismatch)
I20251215 21:06:09.130473 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 96
I20251215 21:06:09.130491 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 112.97
I20251215 21:06:09.130509 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 113 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:09.130879 24095 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251215 21:06:09.132097 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 113 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 111 index: 96. Preceding OpId from leader: term: 113 index: 97. (index mismatch)
I20251215 21:06:09.132498 24095 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251215 21:06:09.292608 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:09.330559 24097 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 113 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:09.330683 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 113 FOLLOWER]: Advancing to term 114
I20251215 21:06:09.331593 24097 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.331817 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 114 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 113. Current term is 114. Ops: []
I20251215 21:06:09.332015 24097 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 114 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:09.332263 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 114 candidate_status { last_received { term: 113 index: 97 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:09.332283 24101 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 113.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251215 21:06:09.332413 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 114 candidate_status { last_received { term: 113 index: 97 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:09.332506 24101 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 LEADER]: Stepping down as leader of term 113
I20251215 21:06:09.332549 24101 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:09.332667 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:09.332700 24101 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 113.97, Last appended by leader: 97, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.332862 24101 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 113 FOLLOWER]: Advancing to term 114
I20251215 21:06:09.332921 24097 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20251215 21:06:09.415726 24103 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 113 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:09.415858 24103 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 113 FOLLOWER]: Advancing to term 114
I20251215 21:06:09.416775 24103 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.417225 24103 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 114 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:09.417600 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 114 candidate_status { last_received { term: 113 index: 97 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:09.417604 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 114 candidate_status { last_received { term: 113 index: 97 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:09.417725 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 114: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:09.418529 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 114.
I20251215 21:06:09.418810 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 114 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:09.418969 24103 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 114 FOLLOWER]: Leader election won for term 114
I20251215 21:06:09.419050 24103 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 114 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:09.419211 24103 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 113.97, Last appended by leader: 97, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.420526 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 113 to 114, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 114 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:09.461297 24097 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 114 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:09.461423 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 114 FOLLOWER]: Advancing to term 115
I20251215 21:06:09.462296 24097 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.462653 24097 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 115 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:09.463090 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 115 candidate_status { last_received { term: 113 index: 97 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:09.463093 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 115 candidate_status { last_received { term: 113 index: 97 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:09.463198 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 114 FOLLOWER]: Advancing to term 115
I20251215 21:06:09.463909 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 115.
I20251215 21:06:09.464157 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:09.464350 24097 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 FOLLOWER]: Leader election won for term 115
I20251215 21:06:09.464519 24097 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:09.464665 24097 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 113.97, Last appended by leader: 97, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.465973 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 114 to 115, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 115 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:09.509622 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 114. Current term is 115. Ops: []
I20251215 21:06:09.510000 24103 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251215 21:06:09.510138 24103 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 115.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20251215 21:06:09.510195 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 114 LEADER]: Stepping down as leader of term 114
I20251215 21:06:09.510248 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 114 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:09.510365 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 114.98, Last appended by leader: 98, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.510483 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 114 FOLLOWER]: Advancing to term 115
I20251215 21:06:09.577298 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 115 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 113 index: 97. Preceding OpId from leader: term: 115 index: 98. (index mismatch)
I20251215 21:06:09.577703 24097 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251215 21:06:09.579617 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 115 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 114 index: 98. Preceding OpId from leader: term: 115 index: 98. (term mismatch)
I20251215 21:06:09.579681 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 97
I20251215 21:06:09.579705 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 114.98
I20251215 21:06:09.579728 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 115 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:09.580056 24097 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251215 21:06:09.882323 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 115 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:09.882457 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 115 FOLLOWER]: Advancing to term 116
I20251215 21:06:09.883348 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.883759 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 116 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:09.884256 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 116 candidate_status { last_received { term: 115 index: 98 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:09.884256 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 116 candidate_status { last_received { term: 115 index: 98 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:09.884665 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:09.884851 24091 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20251215 21:06:09.896292 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 116 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 115. Current term is 116. Ops: []
I20251215 21:06:09.896646 24097 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 115.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251215 21:06:09.896862 24105 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 LEADER]: Stepping down as leader of term 115
I20251215 21:06:09.896920 24105 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:09.897058 24105 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 115.98, Last appended by leader: 98, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.897190 24105 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 115 FOLLOWER]: Advancing to term 116
I20251215 21:06:09.946521 24101 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 115 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:09.946666 24101 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 115 FOLLOWER]: Advancing to term 116
I20251215 21:06:09.947528 24101 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.947924 24101 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 116 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:09.948302 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 116 candidate_status { last_received { term: 115 index: 98 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:09.948328 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 116 candidate_status { last_received { term: 115 index: 98 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:09.948439 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 116: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:09.949157 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 116 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 116.
I20251215 21:06:09.949410 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 116 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:09.949555 24101 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 116 FOLLOWER]: Leader election won for term 116
I20251215 21:06:09.949692 24101 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 116 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:09.949826 24101 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 115.98, Last appended by leader: 98, Current term: 116, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:09.951128 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 115 to 116, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 116 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:10.031450 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:10.031584 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 116 FOLLOWER]: Advancing to term 117
I20251215 21:06:10.032691 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.033141 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 117 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:10.033519 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 117 candidate_status { last_received { term: 115 index: 98 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:10.033519 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 117 candidate_status { last_received { term: 115 index: 98 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:10.033667 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 116 FOLLOWER]: Advancing to term 117
I20251215 21:06:10.034420 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 117.
I20251215 21:06:10.034684 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:10.034847 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 117 FOLLOWER]: Leader election won for term 117
I20251215 21:06:10.035006 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 117 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:10.035156 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 115.98, Last appended by leader: 98, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.036463 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 116 to 117, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 117 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:10.051046 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 117 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 116. Current term is 117. Ops: []
I20251215 21:06:10.051427 24101 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251215 21:06:10.051608 24101 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 116 LEADER]: Stepping down as leader of term 116
I20251215 21:06:10.051651 24101 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 116 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:10.051787 24101 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 116.99, Last appended by leader: 99, Current term: 116, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.051923 24101 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 116 FOLLOWER]: Advancing to term 117
I20251215 21:06:10.121716 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 117 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 116 index: 99. Preceding OpId from leader: term: 117 index: 99. (term mismatch)
I20251215 21:06:10.121819 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 98
I20251215 21:06:10.121836 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 116.99
I20251215 21:06:10.121853 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 117 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:10.122359 24091 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251215 21:06:10.124460 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 117 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 115 index: 98. Preceding OpId from leader: term: 117 index: 99. (index mismatch)
I20251215 21:06:10.124922 24108 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251215 21:06:10.225878 24105 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 117 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:10.226024 24105 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 117 FOLLOWER]: Advancing to term 118
I20251215 21:06:10.226924 24105 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.227319 24105 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 118 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:10.227717 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 118 candidate_status { last_received { term: 117 index: 99 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:10.227717 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 118 candidate_status { last_received { term: 117 index: 99 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:10.227901 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 117 FOLLOWER]: Advancing to term 118
I20251215 21:06:10.228645 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 118.
I20251215 21:06:10.228894 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 118 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:10.229038 24105 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 118 FOLLOWER]: Leader election won for term 118
I20251215 21:06:10.229133 24105 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 118 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:10.229280 24105 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 117.99, Last appended by leader: 99, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.230598 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 117 to 118, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 118 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:10.239200 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 118 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 117. Current term is 118. Ops: []
I20251215 21:06:10.239553 24108 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 117.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251215 21:06:10.239710 24108 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 117 LEADER]: Stepping down as leader of term 117
I20251215 21:06:10.239754 24108 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:10.239866 24108 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 117.99, Last appended by leader: 99, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.239995 24108 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 117 FOLLOWER]: Advancing to term 118
I20251215 21:06:10.292915 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:10.334906 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 118 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 117 index: 99. Preceding OpId from leader: term: 118 index: 100. (index mismatch)
I20251215 21:06:10.335363 24105 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251215 21:06:10.337555 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 118 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 117 index: 99. Preceding OpId from leader: term: 118 index: 100. (index mismatch)
I20251215 21:06:10.337966 24105 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251215 21:06:10.437841 24101 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 118 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:10.437980 24101 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 118 FOLLOWER]: Advancing to term 119
I20251215 21:06:10.438843 24101 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.439244 24101 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 119 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:10.439625 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 119 candidate_status { last_received { term: 118 index: 100 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:10.439625 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 119 candidate_status { last_received { term: 118 index: 100 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:10.440016 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 119 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:10.440169 24101 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 119 FOLLOWER]: Leader election lost for term 119. Reason: could not achieve majority
I20251215 21:06:10.450764 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 119 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 118. Current term is 119. Ops: []
I20251215 21:06:10.455646 24105 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 118.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251215 21:06:10.455842 24097 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 118 LEADER]: Stepping down as leader of term 118
I20251215 21:06:10.455912 24097 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 118 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:10.456053 24097 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 118.100, Last appended by leader: 100, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.456207 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 118 FOLLOWER]: Advancing to term 119
I20251215 21:06:10.528615 24108 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 118 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:10.528776 24108 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 118 FOLLOWER]: Advancing to term 119
I20251215 21:06:10.530104 24108 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.530608 24108 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 119 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:10.530966 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 119 candidate_status { last_received { term: 118 index: 100 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:10.530988 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 119 candidate_status { last_received { term: 118 index: 100 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:10.531097 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 119 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 119: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:10.531993 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 119.
I20251215 21:06:10.532284 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:10.532488 24108 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 119 FOLLOWER]: Leader election won for term 119
I20251215 21:06:10.532580 24108 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 119 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:10.532723 24108 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 118.100, Last appended by leader: 100, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.534109 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 118 to 119, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 119 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:10.574942 24101 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 119 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:10.575074 24101 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 119 FOLLOWER]: Advancing to term 120
I20251215 21:06:10.575940 24101 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.576335 24101 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 120 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:10.576700 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 120 candidate_status { last_received { term: 118 index: 100 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:10.576725 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 120 candidate_status { last_received { term: 118 index: 100 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:10.576874 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 119 FOLLOWER]: Advancing to term 120
I20251215 21:06:10.577594 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 120.
I20251215 21:06:10.577857 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:10.578038 24101 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 120 FOLLOWER]: Leader election won for term 120
I20251215 21:06:10.578198 24101 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 120 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:10.578330 24101 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 118.100, Last appended by leader: 100, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.579562 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 119 to 120, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 120 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:10.614281 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 120 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 119. Current term is 120. Ops: []
I20251215 21:06:10.614696 24108 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251215 21:06:10.614876 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 119 LEADER]: Stepping down as leader of term 119
I20251215 21:06:10.614951 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 119 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:10.615093 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 119.101, Last appended by leader: 101, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.615242 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 119 FOLLOWER]: Advancing to term 120
I20251215 21:06:10.669000 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 120 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 118 index: 100. Preceding OpId from leader: term: 120 index: 101. (index mismatch)
I20251215 21:06:10.669421 24101 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251215 21:06:10.671353 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 120 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 119 index: 101. Preceding OpId from leader: term: 120 index: 101. (term mismatch)
I20251215 21:06:10.671419 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 100
I20251215 21:06:10.671440 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 119.101
I20251215 21:06:10.671471 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 120 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:10.671813 24111 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251215 21:06:10.945899 24097 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 120 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:10.946039 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 120 FOLLOWER]: Advancing to term 121
I20251215 21:06:10.946918 24097 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.947335 24097 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 121 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:10.947747 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 121 candidate_status { last_received { term: 120 index: 101 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:10.947754 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 121 candidate_status { last_received { term: 120 index: 101 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:10.948112 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 121 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:10.948279 24097 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 121 FOLLOWER]: Leader election lost for term 121. Reason: could not achieve majority
I20251215 21:06:10.961107 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 121 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 120. Current term is 121. Ops: []
I20251215 21:06:10.961527 24110 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 120.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251215 21:06:10.961719 24111 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 120 LEADER]: Stepping down as leader of term 120
I20251215 21:06:10.961796 24111 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 120 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:10.961954 24111 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 120.101, Last appended by leader: 101, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.962124 24111 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 120 FOLLOWER]: Advancing to term 121
I20251215 21:06:10.983326 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 120 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:10.983449 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 120 FOLLOWER]: Advancing to term 121
I20251215 21:06:10.984299 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.984680 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 121 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:10.985085 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 121 candidate_status { last_received { term: 120 index: 101 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:10.985101 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 121 candidate_status { last_received { term: 120 index: 101 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:10.985215 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 121: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:10.985848 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 121.
I20251215 21:06:10.986131 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 121 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:10.986321 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 121 FOLLOWER]: Leader election won for term 121
I20251215 21:06:10.986423 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 121 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:10.986577 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 120.101, Last appended by leader: 101, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:10.987874 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 120 to 121, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 121 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:11.060820 24097 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 121 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:11.060976 24097 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 121 FOLLOWER]: Advancing to term 122
I20251215 21:06:11.061893 24097 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.062301 24097 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 122 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:11.062673 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 122 candidate_status { last_received { term: 120 index: 101 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:11.062673 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 122 candidate_status { last_received { term: 120 index: 101 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:11.062824 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 121 FOLLOWER]: Advancing to term 122
I20251215 21:06:11.063572 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 122.
I20251215 21:06:11.063791 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:11.063921 24097 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 122 FOLLOWER]: Leader election won for term 122
I20251215 21:06:11.064085 24097 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 122 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:11.064218 24097 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 120.101, Last appended by leader: 101, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.065490 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 121 to 122, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 122 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:11.089066 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 122 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 121. Current term is 122. Ops: []
I20251215 21:06:11.089560 24091 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251215 21:06:11.089732 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 121 LEADER]: Stepping down as leader of term 121
I20251215 21:06:11.089776 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 121 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:11.089906 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 121.102, Last appended by leader: 102, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.090051 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 121 FOLLOWER]: Advancing to term 122
I20251215 21:06:11.171756 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 122 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 120 index: 101. Preceding OpId from leader: term: 122 index: 102. (index mismatch)
I20251215 21:06:11.172154 24097 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251215 21:06:11.174113 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 122 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 121 index: 102. Preceding OpId from leader: term: 122 index: 102. (term mismatch)
I20251215 21:06:11.174178 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 101
I20251215 21:06:11.174204 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 121.102
I20251215 21:06:11.174228 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 122 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:11.174726 24097 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251215 21:06:11.275244 24111 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 122 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:11.275357 24111 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 122 FOLLOWER]: Advancing to term 123
I20251215 21:06:11.276189 24111 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.276531 24111 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 123 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:11.276844 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 123 candidate_status { last_received { term: 122 index: 102 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:11.276906 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 123 candidate_status { last_received { term: 122 index: 102 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:11.277242 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:11.277397 24111 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20251215 21:06:11.293210 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:11.298264 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 123 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 122. Current term is 123. Ops: []
I20251215 21:06:11.298645 24097 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 122.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251215 21:06:11.298856 24113 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 122 LEADER]: Stepping down as leader of term 122
I20251215 21:06:11.298916 24113 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 122 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:11.299079 24113 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 122.102, Last appended by leader: 102, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.299265 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 122 FOLLOWER]: Advancing to term 123
I20251215 21:06:11.369577 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 122 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:11.369712 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 122 FOLLOWER]: Advancing to term 123
I20251215 21:06:11.370604 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.371006 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 123 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:11.371421 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 123 candidate_status { last_received { term: 122 index: 102 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:11.371443 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 123 candidate_status { last_received { term: 122 index: 102 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:11.371620 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 123: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:11.372188 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 123 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 123.
I20251215 21:06:11.372421 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 123 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:11.372560 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 123 FOLLOWER]: Leader election won for term 123
I20251215 21:06:11.372692 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 123 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:11.372867 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 122.102, Last appended by leader: 102, Current term: 123, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.374257 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 122 to 123, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 123 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:11.378854 24111 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 123 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:11.378923 24111 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 123 FOLLOWER]: Advancing to term 124
I20251215 21:06:11.379578 24111 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.379848 24111 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 124 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:11.380126 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 124 candidate_status { last_received { term: 122 index: 102 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:11.380131 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 124 candidate_status { last_received { term: 122 index: 102 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:11.380257 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 123 FOLLOWER]: Advancing to term 124
I20251215 21:06:11.380946 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 124.
I20251215 21:06:11.381184 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:11.381333 24111 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 124 FOLLOWER]: Leader election won for term 124
I20251215 21:06:11.381412 24111 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 124 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:11.381527 24111 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 122.102, Last appended by leader: 102, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.382680 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 123 to 124, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 124 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:11.486541 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 124 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 123. Current term is 124. Ops: []
I20251215 21:06:11.486965 24091 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251215 21:06:11.487138 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 123 LEADER]: Stepping down as leader of term 123
I20251215 21:06:11.487181 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 123 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:11.487308 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 123.103, Last appended by leader: 103, Current term: 123, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.487429 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 123 FOLLOWER]: Advancing to term 124
I20251215 21:06:11.497092 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 124 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 122 index: 102. Preceding OpId from leader: term: 124 index: 103. (index mismatch)
I20251215 21:06:11.497483 24111 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251215 21:06:11.498739 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 124 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 123 index: 103. Preceding OpId from leader: term: 124 index: 103. (term mismatch)
I20251215 21:06:11.498801 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 102
I20251215 21:06:11.498824 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 123.103
I20251215 21:06:11.498853 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 124 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:11.499311 24110 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251215 21:06:11.600085 24113 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 124 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:11.600224 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 124 FOLLOWER]: Advancing to term 125
I20251215 21:06:11.601183 24113 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.601574 24113 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 125 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:11.601979 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 125 candidate_status { last_received { term: 124 index: 103 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:11.601980 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 125 candidate_status { last_received { term: 124 index: 103 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:11.602097 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 124 FOLLOWER]: Advancing to term 125
I20251215 21:06:11.602830 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 125.
I20251215 21:06:11.603065 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:11.603220 24113 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 FOLLOWER]: Leader election won for term 125
I20251215 21:06:11.603302 24113 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:11.603441 24113 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 124.103, Last appended by leader: 103, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.604707 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 124 to 125, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 125 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:11.617978 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 LEADER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 124. Current term is 125. Ops: []
I20251215 21:06:11.618326 24110 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 124.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251215 21:06:11.618431 24110 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 125.104 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 103
I20251215 21:06:11.618481 24111 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 124 LEADER]: Stepping down as leader of term 124
I20251215 21:06:11.618546 24111 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 124 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:11.618678 24111 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 124.103, Last appended by leader: 103, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.618830 24111 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 124 FOLLOWER]: Advancing to term 125
I20251215 21:06:11.712716 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 125 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 124 index: 103. Preceding OpId from leader: term: 125 index: 104. (index mismatch)
I20251215 21:06:11.712716 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 125 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 124 index: 103. Preceding OpId from leader: term: 125 index: 104. (index mismatch)
I20251215 21:06:11.713153 24113 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251215 21:06:11.713307 24097 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251215 21:06:11.816329 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 125 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:11.816455 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 125 FOLLOWER]: Advancing to term 126
I20251215 21:06:11.817363 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.817742 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 126 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:11.818120 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 126 candidate_status { last_received { term: 125 index: 104 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:11.818123 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 126 candidate_status { last_received { term: 125 index: 104 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:11.818488 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 126 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:11.818634 24091 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 126 FOLLOWER]: Leader election lost for term 126. Reason: could not achieve majority
I20251215 21:06:11.836490 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 126 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 125. Current term is 126. Ops: []
I20251215 21:06:11.836880 24112 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 125.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251215 21:06:11.837061 24113 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 LEADER]: Stepping down as leader of term 125
I20251215 21:06:11.837116 24113 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:11.837237 24113 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 125.104, Last appended by leader: 104, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.837368 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 125 FOLLOWER]: Advancing to term 126
I20251215 21:06:11.892159 24111 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 125 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:11.892300 24111 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 125 FOLLOWER]: Advancing to term 126
I20251215 21:06:11.893236 24111 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.893636 24111 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 126 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:11.894026 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 126 candidate_status { last_received { term: 125 index: 104 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:11.894029 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 126 candidate_status { last_received { term: 125 index: 104 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:11.894152 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 126 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 126: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:11.894893 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 126.
I20251215 21:06:11.895128 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:11.895298 24111 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 126 FOLLOWER]: Leader election won for term 126
I20251215 21:06:11.895390 24111 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 126 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:11.895526 24111 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 125.104, Last appended by leader: 104, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.896773 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 125 to 126, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 126 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:11.941148 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 126 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:11.941283 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 126 FOLLOWER]: Advancing to term 127
I20251215 21:06:11.942178 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.942587 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 127 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:11.943050 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 127 candidate_status { last_received { term: 125 index: 104 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:11.943050 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 127 candidate_status { last_received { term: 125 index: 104 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:11.943225 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 126 FOLLOWER]: Advancing to term 127
I20251215 21:06:11.944057 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 127.
I20251215 21:06:11.944326 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:11.944499 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 127 FOLLOWER]: Leader election won for term 127
I20251215 21:06:11.944586 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 127 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:11.944731 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 125.104, Last appended by leader: 104, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:11.946120 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 126 to 127, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 127 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:12.004997 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 127 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 126. Current term is 127. Ops: []
I20251215 21:06:12.005367 24111 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251215 21:06:12.005545 24110 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 126 LEADER]: Stepping down as leader of term 126
I20251215 21:06:12.005599 24110 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 126 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:12.005751 24110 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 126.105, Last appended by leader: 105, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.005901 24110 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 126 FOLLOWER]: Advancing to term 127
I20251215 21:06:12.044252 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 127 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 126 index: 105. Preceding OpId from leader: term: 127 index: 105. (term mismatch)
I20251215 21:06:12.044337 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 104
I20251215 21:06:12.044354 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 126.105
I20251215 21:06:12.044371 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 127 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:12.044718 24091 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251215 21:06:12.046931 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 127 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 125 index: 104. Preceding OpId from leader: term: 127 index: 105. (index mismatch)
I20251215 21:06:12.047326 24117 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251215 21:06:12.149094 24113 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 127 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:12.149231 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 127 FOLLOWER]: Advancing to term 128
I20251215 21:06:12.150089 24113 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.150466 24113 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 128 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:12.150899 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 128 candidate_status { last_received { term: 127 index: 105 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:12.150899 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 128 candidate_status { last_received { term: 127 index: 105 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:12.151052 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 127 FOLLOWER]: Advancing to term 128
I20251215 21:06:12.151862 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 128 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 128.
I20251215 21:06:12.152096 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 128 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:12.152277 24113 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 128 FOLLOWER]: Leader election won for term 128
I20251215 21:06:12.152354 24113 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 128 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:12.152491 24113 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 127.105, Last appended by leader: 105, Current term: 128, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.153828 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 127 to 128, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 128 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:12.158326 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 128 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 127. Current term is 128. Ops: []
I20251215 21:06:12.158658 24117 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 127.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251215 21:06:12.158783 24117 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 127 LEADER]: Stepping down as leader of term 127
I20251215 21:06:12.158826 24117 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 127 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:12.158931 24117 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 127.105, Last appended by leader: 105, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.159050 24117 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 127 FOLLOWER]: Advancing to term 128
I20251215 21:06:12.239957 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 128 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 127 index: 105. Preceding OpId from leader: term: 128 index: 106. (index mismatch)
I20251215 21:06:12.240334 24113 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251215 21:06:12.241672 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 128 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 127 index: 105. Preceding OpId from leader: term: 128 index: 106. (index mismatch)
I20251215 21:06:12.242141 24113 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251215 21:06:12.293503 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:12.505874 24110 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 128 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:12.505986 24110 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 128 FOLLOWER]: Advancing to term 129
I20251215 21:06:12.506826 24110 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.507185 24110 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 129 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:12.507462 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 129 candidate_status { last_received { term: 128 index: 106 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:12.507573 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 129 candidate_status { last_received { term: 128 index: 106 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:12.507884 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 129 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:12.508031 24110 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 129 FOLLOWER]: Leader election lost for term 129. Reason: could not achieve majority
I20251215 21:06:12.530395 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 129 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 128. Current term is 129. Ops: []
I20251215 21:06:12.530791 24118 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 128.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251215 21:06:12.530978 24113 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 128 LEADER]: Stepping down as leader of term 128
I20251215 21:06:12.531054 24113 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 128 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:12.531215 24113 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 128.106, Last appended by leader: 106, Current term: 128, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.531379 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 128 FOLLOWER]: Advancing to term 129
I20251215 21:06:12.604314 24117 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 128 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:12.604460 24117 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 128 FOLLOWER]: Advancing to term 129
I20251215 21:06:12.605491 24117 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.605906 24117 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 129 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:12.606335 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 129 candidate_status { last_received { term: 128 index: 106 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:12.606336 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 129 candidate_status { last_received { term: 128 index: 106 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:12.606500 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 129: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:12.607199 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 129.
I20251215 21:06:12.607481 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:12.607638 24117 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 129 FOLLOWER]: Leader election won for term 129
I20251215 21:06:12.607723 24117 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 129 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:12.607869 24117 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 128.106, Last appended by leader: 106, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.609278 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 128 to 129, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 129 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:12.650014 24110 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 129 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:12.650172 24110 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 129 FOLLOWER]: Advancing to term 130
I20251215 21:06:12.651108 24110 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.651546 24110 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 130 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:12.651943 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 130 candidate_status { last_received { term: 128 index: 106 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:12.651943 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 130 candidate_status { last_received { term: 128 index: 106 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:12.652089 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 129 FOLLOWER]: Advancing to term 130
I20251215 21:06:12.652787 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 130.
I20251215 21:06:12.653045 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:12.653218 24110 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 FOLLOWER]: Leader election won for term 130
I20251215 21:06:12.653374 24110 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:12.653515 24110 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 128.106, Last appended by leader: 106, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.654767 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 129 to 130, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 130 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:12.693316 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 130 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 129. Current term is 130. Ops: []
I20251215 21:06:12.693723 24117 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251215 21:06:12.693902 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 129 LEADER]: Stepping down as leader of term 129
I20251215 21:06:12.693961 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 129. Current term is 130. Ops: []
I20251215 21:06:12.693970 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 129 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:12.694136 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 129.107, Last appended by leader: 107, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.694306 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 129 FOLLOWER]: Advancing to term 130
W20251215 21:06:12.694303 24117 consensus_queue.cc:1175] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ec6828f7b4fb41348830bc708ea09c2b" responder_term: 130 status { last_received { term: 130 index: 107 } last_committed_idx: 106 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 129. Current term is 130. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251215 21:06:12.764205 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 130 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 128 index: 106. Preceding OpId from leader: term: 130 index: 107. (index mismatch)
I20251215 21:06:12.764598 24110 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251215 21:06:12.766520 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 130 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 129 index: 107. Preceding OpId from leader: term: 130 index: 107. (term mismatch)
I20251215 21:06:12.766587 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 106
I20251215 21:06:12.766610 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 129.107
I20251215 21:06:12.766633 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 130 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:12.766965 24120 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251215 21:06:12.867116 24113 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 130 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:12.867239 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 130 FOLLOWER]: Advancing to term 131
I20251215 21:06:12.868068 24113 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.868433 24113 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 131 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:12.868738 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 131 candidate_status { last_received { term: 130 index: 107 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:12.868738 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 131 candidate_status { last_received { term: 130 index: 107 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:12.869167 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 131 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:12.869328 24113 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 131 FOLLOWER]: Leader election lost for term 131. Reason: could not achieve majority
I20251215 21:06:12.884972 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 131 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 130. Current term is 131. Ops: []
I20251215 21:06:12.885335 24120 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 130.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251215 21:06:12.885515 24119 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 LEADER]: Stepping down as leader of term 130
I20251215 21:06:12.885569 24119 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:12.885681 24119 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 130.107, Last appended by leader: 107, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.885808 24119 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 130 FOLLOWER]: Advancing to term 131
I20251215 21:06:12.952077 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 130 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:12.952199 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 130 FOLLOWER]: Advancing to term 131
I20251215 21:06:12.953096 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.953466 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 131 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:12.953850 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 131 candidate_status { last_received { term: 130 index: 107 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:12.953847 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 131 candidate_status { last_received { term: 130 index: 107 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:12.954025 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 131: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:12.954716 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 131.
I20251215 21:06:12.954970 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:12.955121 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 131 FOLLOWER]: Leader election won for term 131
I20251215 21:06:12.955199 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 131 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:12.955335 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 130.107, Last appended by leader: 107, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:12.956724 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 130 to 131, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 131 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:13.018508 24113 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 131 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:13.018636 24113 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 131 FOLLOWER]: Advancing to term 132
I20251215 21:06:13.019493 24113 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.019882 24113 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 132 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:13.020249 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 132 candidate_status { last_received { term: 130 index: 107 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:13.020303 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 132 candidate_status { last_received { term: 130 index: 107 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:13.020439 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 131 FOLLOWER]: Advancing to term 132
I20251215 21:06:13.021227 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 132.
I20251215 21:06:13.021484 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:13.021649 24113 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 132 FOLLOWER]: Leader election won for term 132
I20251215 21:06:13.021732 24113 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 132 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:13.021878 24113 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 130.107, Last appended by leader: 107, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.023181 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 131 to 132, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 132 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:13.076820 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 132 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 131. Current term is 132. Ops: []
I20251215 21:06:13.077230 24091 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251215 21:06:13.077404 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 131 LEADER]: Stepping down as leader of term 131
I20251215 21:06:13.077448 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:13.077557 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 131.108, Last appended by leader: 108, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.077677 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 131 FOLLOWER]: Advancing to term 132
I20251215 21:06:13.113440 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 132 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 130 index: 107. Preceding OpId from leader: term: 132 index: 108. (index mismatch)
I20251215 21:06:13.113850 24113 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251215 21:06:13.115911 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 132 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 131 index: 108. Preceding OpId from leader: term: 132 index: 108. (term mismatch)
I20251215 21:06:13.115975 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 107
I20251215 21:06:13.115999 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 131.108
I20251215 21:06:13.116021 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 132 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:13.116292 24113 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251215 21:06:13.293823 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:13.311950 24119 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 132 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:13.312053 24119 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 132 FOLLOWER]: Advancing to term 133
I20251215 21:06:13.312965 24119 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.313349 24119 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 133 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:13.313652 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 133 candidate_status { last_received { term: 132 index: 108 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:13.313690 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 133 candidate_status { last_received { term: 132 index: 108 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:13.314040 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:13.314200 24119 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20251215 21:06:13.318604 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 133 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 132. Current term is 133. Ops: []
I20251215 21:06:13.318951 24113 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 132.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251215 21:06:13.319118 24118 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 132 LEADER]: Stepping down as leader of term 132
I20251215 21:06:13.319180 24118 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 132 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:13.319306 24118 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 132.108, Last appended by leader: 108, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.319442 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 132 FOLLOWER]: Advancing to term 133
I20251215 21:06:13.410209 24117 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 132 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:13.410339 24117 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 132 FOLLOWER]: Advancing to term 133
I20251215 21:06:13.411193 24117 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.411566 24117 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 133 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:13.411964 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 133 candidate_status { last_received { term: 132 index: 108 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:13.411988 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 133 candidate_status { last_received { term: 132 index: 108 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:13.412158 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 133: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:13.412699 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 133 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 133.
I20251215 21:06:13.412948 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 133 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:13.413126 24117 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 133 FOLLOWER]: Leader election won for term 133
I20251215 21:06:13.413223 24117 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 133 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:13.413376 24117 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 132.108, Last appended by leader: 108, Current term: 133, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.414711 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 132 to 133, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 133 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:13.451704 24119 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 133 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:13.451828 24119 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 133 FOLLOWER]: Advancing to term 134
I20251215 21:06:13.452705 24119 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.453156 24119 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 134 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:13.453526 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 134 candidate_status { last_received { term: 132 index: 108 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:13.453534 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 134 candidate_status { last_received { term: 132 index: 108 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:13.453670 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 133 FOLLOWER]: Advancing to term 134
I20251215 21:06:13.454394 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 134.
I20251215 21:06:13.454645 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:13.454795 24119 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 FOLLOWER]: Leader election won for term 134
I20251215 21:06:13.454974 24119 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:13.455113 24119 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 132.108, Last appended by leader: 108, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.456391 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 133 to 134, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 134 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:13.499701 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 133. Current term is 134. Ops: []
I20251215 21:06:13.500110 24117 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251215 21:06:13.500249 24117 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer ec6828f7b4fb41348830bc708ea09c2b log is divergent from this leader: its last log entry 134.109 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 108
I20251215 21:06:13.500300 24091 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 133 LEADER]: Stepping down as leader of term 133
I20251215 21:06:13.500351 24091 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 133 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:13.500473 24091 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 133.109, Last appended by leader: 109, Current term: 133, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.500644 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 133 FOLLOWER]: Advancing to term 134
I20251215 21:06:13.544320 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 134 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 132 index: 108. Preceding OpId from leader: term: 134 index: 109. (index mismatch)
I20251215 21:06:13.544715 24119 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251215 21:06:13.547243 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 134 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 133 index: 109. Preceding OpId from leader: term: 134 index: 109. (term mismatch)
I20251215 21:06:13.547312 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 108
I20251215 21:06:13.547336 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 133.109
I20251215 21:06:13.547359 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 134 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:13.547708 24122 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251215 21:06:13.740312 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 134 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:13.740458 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 134 FOLLOWER]: Advancing to term 135
I20251215 21:06:13.741413 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.741890 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 135 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:13.742295 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 135 candidate_status { last_received { term: 134 index: 109 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:13.742295 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 135 candidate_status { last_received { term: 134 index: 109 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:13.742692 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 135 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:13.742844 24091 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 135 FOLLOWER]: Leader election lost for term 135. Reason: could not achieve majority
I20251215 21:06:13.745421 24118 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 134 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:13.745491 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 134 FOLLOWER]: Advancing to term 135
I20251215 21:06:13.746244 24118 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.746548 24118 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 135 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:13.746873 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 135 candidate_status { last_received { term: 134 index: 109 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:13.746892 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 135 candidate_status { last_received { term: 134 index: 109 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:13.747017 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 135: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:13.747273 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 135 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:13.747429 24118 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 135 FOLLOWER]: Leader election lost for term 135. Reason: could not achieve majority
I20251215 21:06:13.757879 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 135 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 134. Current term is 135. Ops: []
I20251215 21:06:13.758266 24119 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 134.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251215 21:06:13.758455 24122 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 LEADER]: Stepping down as leader of term 134
I20251215 21:06:13.758509 24122 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:13.758657 24122 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 134.109, Last appended by leader: 109, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.758808 24122 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 134 FOLLOWER]: Advancing to term 135
I20251215 21:06:13.884409 24091 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 135 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:13.884555 24091 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 135 FOLLOWER]: Advancing to term 136
I20251215 21:06:13.885515 24091 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.885923 24091 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 136 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:13.886348 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 136 candidate_status { last_received { term: 134 index: 109 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:13.886355 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 136 candidate_status { last_received { term: 134 index: 109 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:13.886457 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 135 FOLLOWER]: Advancing to term 136
I20251215 21:06:13.886492 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 135 FOLLOWER]: Advancing to term 136
I20251215 21:06:13.887177 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 136.
I20251215 21:06:13.887231 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 136.
I20251215 21:06:13.887452 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:06:13.887614 24091 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 136 FOLLOWER]: Leader election won for term 136
I20251215 21:06:13.887696 24091 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 136 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:13.887827 24091 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 134.109, Last appended by leader: 109, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:13.889175 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 134 to 136, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 136 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:13.976218 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 136 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 134 index: 109. Preceding OpId from leader: term: 136 index: 110. (index mismatch)
I20251215 21:06:13.976595 24091 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251215 21:06:13.978724 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 136 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 134 index: 109. Preceding OpId from leader: term: 136 index: 110. (index mismatch)
I20251215 21:06:13.979066 24124 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251215 21:06:14.079433 24122 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 136 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:14.079553 24122 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 136 FOLLOWER]: Advancing to term 137
I20251215 21:06:14.080458 24122 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.080880 24122 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 137 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:14.081213 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 137 candidate_status { last_received { term: 136 index: 110 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:14.081215 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 137 candidate_status { last_received { term: 136 index: 110 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:14.081599 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:14.081760 24122 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I20251215 21:06:14.092976 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 137 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 136. Current term is 137. Ops: []
I20251215 21:06:14.093402 24124 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 136.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251215 21:06:14.093598 24117 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 136 LEADER]: Stepping down as leader of term 136
I20251215 21:06:14.093657 24117 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:14.093776 24117 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 136.110, Last appended by leader: 110, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.093919 24117 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 136 FOLLOWER]: Advancing to term 137
I20251215 21:06:14.168735 24118 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 136 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:14.168875 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 136 FOLLOWER]: Advancing to term 137
I20251215 21:06:14.169739 24118 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.170105 24118 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 137 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:14.170529 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 137 candidate_status { last_received { term: 136 index: 110 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:14.170542 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 137 candidate_status { last_received { term: 136 index: 110 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:14.170670 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate bf42b6838bde4b4199f10f61556786ef in current term 137: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:14.171473 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 137.
I20251215 21:06:14.171756 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:14.171916 24118 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 137 FOLLOWER]: Leader election won for term 137
I20251215 21:06:14.172075 24118 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 137 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:14.172219 24118 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 136.110, Last appended by leader: 110, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.173501 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 136 to 137, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 137 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:14.229640 24122 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 137 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:14.229765 24122 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 137 FOLLOWER]: Advancing to term 138
I20251215 21:06:14.230665 24122 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.231060 24122 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 138 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:14.231410 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 138 candidate_status { last_received { term: 136 index: 110 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:14.231499 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 137 FOLLOWER]: Advancing to term 138
I20251215 21:06:14.231513 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 138 candidate_status { last_received { term: 136 index: 110 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:14.232187 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 138.
I20251215 21:06:14.232415 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:14.232563 24122 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 138 FOLLOWER]: Leader election won for term 138
I20251215 21:06:14.232663 24122 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 138 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:14.232823 24122 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 136.110, Last appended by leader: 110, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.234179 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 137 to 138, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 138 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:14.259598 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 138 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 137. Current term is 138. Ops: []
I20251215 21:06:14.259975 24118 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251215 21:06:14.260149 24118 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 137 LEADER]: Stepping down as leader of term 137
I20251215 21:06:14.260197 24118 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 137 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:14.260334 24118 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 137.111, Last appended by leader: 111, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.260473 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 137 FOLLOWER]: Advancing to term 138
I20251215 21:06:14.294133 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:14.316296 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 138 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 137 index: 111. Preceding OpId from leader: term: 138 index: 111. (term mismatch)
I20251215 21:06:14.316380 23751 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting all ops after (but not including) 110
I20251215 21:06:14.316396 23751 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: Aborting uncommitted NO_OP operation due to leader change: 137.111
I20251215 21:06:14.316414 23751 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 138 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:14.316779 24122 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251215 21:06:14.318851 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 138 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 136 index: 110. Preceding OpId from leader: term: 138 index: 111. (index mismatch)
I20251215 21:06:14.319310 24119 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251215 21:06:14.604314 24118 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 138 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:14.604445 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 138 FOLLOWER]: Advancing to term 139
I20251215 21:06:14.605401 24118 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.605784 24118 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 139 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:14.606199 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 139 candidate_status { last_received { term: 138 index: 111 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:14.606292 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 139 candidate_status { last_received { term: 138 index: 111 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:14.606683 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:14.606846 24118 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20251215 21:06:14.624478 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 139 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 138. Current term is 139. Ops: []
I20251215 21:06:14.624861 24126 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 138.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251215 21:06:14.625052 24119 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 138 LEADER]: Stepping down as leader of term 138
I20251215 21:06:14.625121 24119 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 138 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:14.625270 24119 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 138.111, Last appended by leader: 111, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.625440 24119 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 138 FOLLOWER]: Advancing to term 139
I20251215 21:06:14.685209 24117 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 138 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:14.685369 24117 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 138 FOLLOWER]: Advancing to term 139
I20251215 21:06:14.686338 24117 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.686843 24117 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 139 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:14.687281 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 139 candidate_status { last_received { term: 138 index: 111 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:14.687283 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 139 candidate_status { last_received { term: 138 index: 111 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:14.687433 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 139: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:14.688299 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 139.
I20251215 21:06:14.688597 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:14.688769 24117 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 139 FOLLOWER]: Leader election won for term 139
I20251215 21:06:14.688881 24117 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 139 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:14.689040 24117 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 138.111, Last appended by leader: 111, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.690474 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 138 to 139, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 139 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:14.733031 24118 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:14.733165 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 139 FOLLOWER]: Advancing to term 140
I20251215 21:06:14.734037 24118 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.734438 24118 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 140 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:14.734843 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 140 candidate_status { last_received { term: 138 index: 111 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:14.734839 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 140 candidate_status { last_received { term: 138 index: 111 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:14.734963 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 139 FOLLOWER]: Advancing to term 140
I20251215 21:06:14.735704 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 140.
I20251215 21:06:14.735981 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 140 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:14.736191 24118 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 FOLLOWER]: Leader election won for term 140
I20251215 21:06:14.736285 24118 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:14.736424 24118 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 138.111, Last appended by leader: 111, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.737848 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 139 to 140, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 140 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:14.768466 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 LEADER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 139. Current term is 140. Ops: []
I20251215 21:06:14.768884 24117 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251215 21:06:14.769023 24117 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 140.112 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 111
I20251215 21:06:14.769078 24124 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 139 LEADER]: Stepping down as leader of term 139
I20251215 21:06:14.769132 24124 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:14.769258 24124 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 139.112, Last appended by leader: 112, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.769380 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 139 FOLLOWER]: Advancing to term 140
I20251215 21:06:14.818876 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 140 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 139 index: 112. Preceding OpId from leader: term: 140 index: 112. (term mismatch)
I20251215 21:06:14.818974 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 111
I20251215 21:06:14.818997 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 139.112
I20251215 21:06:14.819023 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:14.819420 24118 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251215 21:06:14.821858 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 140 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 138 index: 111. Preceding OpId from leader: term: 140 index: 112. (index mismatch)
I20251215 21:06:14.822219 24118 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251215 21:06:14.923754 24119 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 140 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:14.923887 24119 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 140 FOLLOWER]: Advancing to term 141
I20251215 21:06:14.924768 24119 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.925177 24119 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 141 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:14.925539 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 141 candidate_status { last_received { term: 140 index: 112 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:14.925556 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 141 candidate_status { last_received { term: 140 index: 112 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:14.925889 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:14.926054 24119 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I20251215 21:06:14.928457 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 141 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 140. Current term is 141. Ops: []
I20251215 21:06:14.928778 24127 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 140.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251215 21:06:14.928948 24125 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 LEADER]: Stepping down as leader of term 140
I20251215 21:06:14.929013 24125 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:14.929131 24125 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 140.112, Last appended by leader: 112, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:14.929270 24125 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 140 FOLLOWER]: Advancing to term 141
I20251215 21:06:15.002157 24124 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 140 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:15.002300 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 140 FOLLOWER]: Advancing to term 141
I20251215 21:06:15.003187 24124 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.003610 24124 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 141 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:15.004045 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 141 candidate_status { last_received { term: 140 index: 112 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:15.004041 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 141 candidate_status { last_received { term: 140 index: 112 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:15.004195 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 141: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:15.004850 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 141.
I20251215 21:06:15.005087 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:15.005237 24124 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 141 FOLLOWER]: Leader election won for term 141
I20251215 21:06:15.005318 24124 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 141 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:15.005455 24124 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 140.112, Last appended by leader: 112, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.006718 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 140 to 141, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 141 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:15.036507 24119 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:15.036629 24119 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 141 FOLLOWER]: Advancing to term 142
I20251215 21:06:15.037530 24119 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.037941 24119 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 142 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:15.038301 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 142 candidate_status { last_received { term: 140 index: 112 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:15.038301 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 142 candidate_status { last_received { term: 140 index: 112 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:15.038414 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 141 FOLLOWER]: Advancing to term 142
I20251215 21:06:15.039137 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 142.
I20251215 21:06:15.039390 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:15.039552 24119 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 142 FOLLOWER]: Leader election won for term 142
I20251215 21:06:15.039642 24119 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 142 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:15.039775 24119 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 140.112, Last appended by leader: 112, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.041079 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 141 to 142, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 142 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:15.099529 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 142 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 141. Current term is 142. Ops: []
I20251215 21:06:15.099946 24124 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251215 21:06:15.100116 24124 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 141 LEADER]: Stepping down as leader of term 141
I20251215 21:06:15.100160 24124 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 141 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:15.100272 24124 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 141.113, Last appended by leader: 113, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.100394 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 141 FOLLOWER]: Advancing to term 142
I20251215 21:06:15.130122 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 142 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 141 index: 113. Preceding OpId from leader: term: 142 index: 113. (term mismatch)
I20251215 21:06:15.130204 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 112
I20251215 21:06:15.130221 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 141.113
I20251215 21:06:15.130239 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 142 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:15.130563 24119 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251215 21:06:15.132586 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 142 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 140 index: 112. Preceding OpId from leader: term: 142 index: 113. (index mismatch)
I20251215 21:06:15.133090 24128 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251215 21:06:15.234782 24125 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 142 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:15.234925 24125 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 142 FOLLOWER]: Advancing to term 143
I20251215 21:06:15.235807 24125 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.236196 24125 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 143 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:15.236577 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 143 candidate_status { last_received { term: 142 index: 113 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:15.236584 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 143 candidate_status { last_received { term: 142 index: 113 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:15.236677 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 142 FOLLOWER]: Advancing to term 143
I20251215 21:06:15.237442 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 143.
I20251215 21:06:15.237677 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 143 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:15.237839 24125 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 143 FOLLOWER]: Leader election won for term 143
I20251215 21:06:15.237913 24125 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 143 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:15.238051 24125 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 142.113, Last appended by leader: 113, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.239287 23645 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 142 to 143, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 143 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:15.242208 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 143 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 142. Current term is 143. Ops: []
I20251215 21:06:15.242514 24128 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 142.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20251215 21:06:15.242652 24128 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 142 LEADER]: Stepping down as leader of term 142
I20251215 21:06:15.242691 24128 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 142 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:15.242820 24128 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 142.113, Last appended by leader: 113, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.242964 24128 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 142 FOLLOWER]: Advancing to term 143
I20251215 21:06:15.294450 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:15.338322 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 143 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 142 index: 113. Preceding OpId from leader: term: 143 index: 114. (index mismatch)
I20251215 21:06:15.338701 24125 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20251215 21:06:15.339356 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 143 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 142 index: 113. Preceding OpId from leader: term: 143 index: 114. (index mismatch)
I20251215 21:06:15.339707 24127 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20251215 21:06:15.441140 24124 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 143 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:15.441262 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 143 FOLLOWER]: Advancing to term 144
I20251215 21:06:15.442143 24124 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.442523 24124 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 144 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:15.442922 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 144 candidate_status { last_received { term: 143 index: 114 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:15.442932 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 144 candidate_status { last_received { term: 143 index: 114 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:15.443298 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 144 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:15.443454 24124 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 144 FOLLOWER]: Leader election lost for term 144. Reason: could not achieve majority
I20251215 21:06:15.464146 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 144 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 143. Current term is 144. Ops: []
I20251215 21:06:15.464504 24127 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 143.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251215 21:06:15.464689 24118 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 143 LEADER]: Stepping down as leader of term 143
I20251215 21:06:15.464756 24118 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 143 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:15.464918 24118 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 143.114, Last appended by leader: 114, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.465075 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 143 FOLLOWER]: Advancing to term 144
I20251215 21:06:15.539661 24128 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 143 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:15.539796 24128 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 143 FOLLOWER]: Advancing to term 144
I20251215 21:06:15.540661 24128 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.541100 24128 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 144 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:15.541509 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 144 candidate_status { last_received { term: 143 index: 114 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:15.541507 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 144 candidate_status { last_received { term: 143 index: 114 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:15.541661 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 144 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 144: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:15.542322 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 144.
I20251215 21:06:15.542551 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:15.542693 24128 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 144 FOLLOWER]: Leader election won for term 144
I20251215 21:06:15.542773 24128 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 144 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:15.542907 24128 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 143.114, Last appended by leader: 114, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.544209 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 143 to 144, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 144 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:15.552863 24124 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 144 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:15.552953 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 144 FOLLOWER]: Advancing to term 145
I20251215 21:06:15.553714 24124 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.554039 24124 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 145 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:15.554335 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 145 candidate_status { last_received { term: 143 index: 114 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:15.554356 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 145 candidate_status { last_received { term: 143 index: 114 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:15.554447 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 144 FOLLOWER]: Advancing to term 145
I20251215 21:06:15.555167 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 145.
I20251215 21:06:15.555413 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 145 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:15.555554 24124 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 FOLLOWER]: Leader election won for term 145
I20251215 21:06:15.555632 24124 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:15.555753 24124 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 143.114, Last appended by leader: 114, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.557040 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 144 to 145, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 145 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:15.629226 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 LEADER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 144. Current term is 145. Ops: []
I20251215 21:06:15.629644 24128 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251215 21:06:15.629777 24128 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer 31ff5cfb5d624764ae78ac08450e9022 log is divergent from this leader: its last log entry 145.115 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 114
I20251215 21:06:15.629829 24126 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 144 LEADER]: Stepping down as leader of term 144
I20251215 21:06:15.629912 24126 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 144 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:15.630061 24126 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 144.115, Last appended by leader: 115, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.630203 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 144 FOLLOWER]: Advancing to term 145
I20251215 21:06:15.632929 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 145 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 144 index: 115. Preceding OpId from leader: term: 145 index: 115. (term mismatch)
I20251215 21:06:15.632983 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 114
I20251215 21:06:15.633008 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 144.115
I20251215 21:06:15.633033 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 145 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:15.633365 24124 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251215 21:06:15.635074 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 145 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 143 index: 114. Preceding OpId from leader: term: 145 index: 115. (index mismatch)
I20251215 21:06:15.635392 24124 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251215 21:06:15.736660 24118 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 145 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:15.736820 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 145 FOLLOWER]: Advancing to term 146
I20251215 21:06:15.737710 24118 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.738109 24118 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 146 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:15.738509 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 146 candidate_status { last_received { term: 145 index: 115 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:15.738513 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 146 candidate_status { last_received { term: 145 index: 115 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:15.738890 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 146 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:15.739048 24118 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 146 FOLLOWER]: Leader election lost for term 146. Reason: could not achieve majority
I20251215 21:06:15.756096 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 146 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 145. Current term is 146. Ops: []
I20251215 21:06:15.756665 24124 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 145.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251215 21:06:15.756851 24124 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 LEADER]: Stepping down as leader of term 145
I20251215 21:06:15.756901 24124 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:15.757022 24124 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 145.115, Last appended by leader: 115, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.757197 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 145 FOLLOWER]: Advancing to term 146
I20251215 21:06:15.852514 24118 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 146 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:15.852638 24118 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 146 FOLLOWER]: Advancing to term 147
I20251215 21:06:15.853538 24118 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.853920 24118 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 147 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:15.854336 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 147 candidate_status { last_received { term: 145 index: 115 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:15.854333 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 147 candidate_status { last_received { term: 145 index: 115 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:15.854456 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 145 FOLLOWER]: Advancing to term 147
I20251215 21:06:15.854471 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 146 FOLLOWER]: Advancing to term 147
I20251215 21:06:15.855223 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 147.
I20251215 21:06:15.855273 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 147.
I20251215 21:06:15.855482 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters:
I20251215 21:06:15.855643 24118 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 147 FOLLOWER]: Leader election won for term 147
I20251215 21:06:15.855710 24118 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 147 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:15.855844 24118 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 145.115, Last appended by leader: 115, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:15.857216 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 145 to 147, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 147 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:15.966271 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 147 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 145 index: 115. Preceding OpId from leader: term: 147 index: 116. (index mismatch)
I20251215 21:06:15.966655 24118 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251215 21:06:15.968900 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 147 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 145 index: 115. Preceding OpId from leader: term: 147 index: 116. (index mismatch)
I20251215 21:06:15.969277 24130 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251215 21:06:16.070902 24126 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 147 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:16.071026 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 147 FOLLOWER]: Advancing to term 148
I20251215 21:06:16.071846 24126 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.072185 24126 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 148 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:16.072485 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 148 candidate_status { last_received { term: 147 index: 116 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:16.072497 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 148 candidate_status { last_received { term: 147 index: 116 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:16.072837 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 148 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:16.072993 24126 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 148 FOLLOWER]: Leader election lost for term 148. Reason: could not achieve majority
I20251215 21:06:16.087769 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 148 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 147. Current term is 148. Ops: []
I20251215 21:06:16.088198 24130 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 147.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251215 21:06:16.088384 24130 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 147 LEADER]: Stepping down as leader of term 147
I20251215 21:06:16.088431 24130 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 147 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:16.088603 24130 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 147.116, Last appended by leader: 116, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.088786 24130 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 147 FOLLOWER]: Advancing to term 148
I20251215 21:06:16.168454 24129 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 147 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:16.168578 24129 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 147 FOLLOWER]: Advancing to term 148
I20251215 21:06:16.169509 24129 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.169881 24129 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 148 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:16.170287 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 148 candidate_status { last_received { term: 147 index: 116 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:16.170287 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 148 candidate_status { last_received { term: 147 index: 116 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:16.170462 23826 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 148: Already voted for candidate ec6828f7b4fb41348830bc708ea09c2b in this term.
I20251215 21:06:16.171129 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 148.
I20251215 21:06:16.171423 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:16.171594 24129 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 148 FOLLOWER]: Leader election won for term 148
I20251215 21:06:16.171679 24129 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 148 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:16.171963 24129 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 147.116, Last appended by leader: 116, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.173257 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 147 to 148, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 148 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:16.199468 24126 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 148 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:16.199602 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 148 FOLLOWER]: Advancing to term 149
I20251215 21:06:16.200495 24126 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.200944 24126 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 149 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:16.201315 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 149 candidate_status { last_received { term: 147 index: 116 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:16.201321 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 149 candidate_status { last_received { term: 147 index: 116 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:16.201449 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 148 FOLLOWER]: Advancing to term 149
I20251215 21:06:16.202190 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 149.
I20251215 21:06:16.202435 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:16.202589 24126 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 149 FOLLOWER]: Leader election won for term 149
I20251215 21:06:16.202750 24126 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 149 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:16.202888 24126 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 147.116, Last appended by leader: 116, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.204162 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 148 to 149, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 149 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:16.277537 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 149 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 148. Current term is 149. Ops: []
I20251215 21:06:16.277946 24129 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251215 21:06:16.278105 24129 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 148 LEADER]: Stepping down as leader of term 148
I20251215 21:06:16.278151 24129 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:16.278282 24129 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 148.117, Last appended by leader: 117, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.278410 24129 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 148 FOLLOWER]: Advancing to term 149
I20251215 21:06:16.294751 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:16.298220 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 149 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 148 index: 117. Preceding OpId from leader: term: 149 index: 117. (term mismatch)
I20251215 21:06:16.298292 23902 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting all ops after (but not including) 116
I20251215 21:06:16.298309 23902 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: Aborting uncommitted NO_OP operation due to leader change: 148.117
I20251215 21:06:16.298327 23902 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 149 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:16.298661 24126 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251215 21:06:16.300361 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 149 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 147 index: 116. Preceding OpId from leader: term: 149 index: 117. (index mismatch)
I20251215 21:06:16.300724 24131 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251215 21:06:16.484894 24130 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 149 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:16.485034 24130 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 149 FOLLOWER]: Advancing to term 150
I20251215 21:06:16.485951 24130 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.486346 24130 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 150 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:16.486742 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 150 candidate_status { last_received { term: 149 index: 117 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:16.486738 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 150 candidate_status { last_received { term: 149 index: 117 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:16.487104 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 150 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:16.487260 24130 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Leader election lost for term 150. Reason: could not achieve majority
I20251215 21:06:16.492331 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 149. Current term is 150. Ops: []
I20251215 21:06:16.492715 24131 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 149.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20251215 21:06:16.492933 24126 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 149 LEADER]: Stepping down as leader of term 149
I20251215 21:06:16.492988 24126 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 149 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:16.493117 24126 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 149.117, Last appended by leader: 117, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.493271 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 149 FOLLOWER]: Advancing to term 150
I20251215 21:06:16.504386 24129 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 149 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:16.504508 24129 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 149 FOLLOWER]: Advancing to term 150
I20251215 21:06:16.505434 24129 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.505818 24129 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 150 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:16.506202 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 150 candidate_status { last_received { term: 149 index: 117 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:16.506202 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 150 candidate_status { last_received { term: 149 index: 117 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:16.506367 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate 31ff5cfb5d624764ae78ac08450e9022 in current term 150: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:16.507040 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 150.
I20251215 21:06:16.507284 23868 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:16.507458 24129 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 150 FOLLOWER]: Leader election won for term 150
I20251215 21:06:16.507540 24129 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 150 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:16.507678 24129 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 149.117, Last appended by leader: 117, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.508915 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 149 to 150, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 150 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:16.589108 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 149 index: 117. Preceding OpId from leader: term: 150 index: 118. (index mismatch)
I20251215 21:06:16.589502 24129 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20251215 21:06:16.591621 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 150 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 149 index: 117. Preceding OpId from leader: term: 150 index: 118. (index mismatch)
I20251215 21:06:16.591922 24133 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20251215 21:06:16.691959 24130 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:16.692080 24130 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 150 FOLLOWER]: Advancing to term 151
I20251215 21:06:16.693001 24130 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.693406 24130 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 151 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:16.693737 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 151 candidate_status { last_received { term: 150 index: 118 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:16.693766 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 151 candidate_status { last_received { term: 150 index: 118 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:16.694072 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 151 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:16.694219 24130 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 151 FOLLOWER]: Leader election lost for term 151. Reason: could not achieve majority
I20251215 21:06:16.698688 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 151 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 150. Current term is 151. Ops: []
I20251215 21:06:16.699002 24133 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 150.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251215 21:06:16.699173 24124 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 150 LEADER]: Stepping down as leader of term 150
I20251215 21:06:16.699229 24124 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 150 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:16.699350 24124 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 150.118, Last appended by leader: 118, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.699493 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 150 FOLLOWER]: Advancing to term 151
I20251215 21:06:16.783954 24126 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 150 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:16.784101 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 150 FOLLOWER]: Advancing to term 151
I20251215 21:06:16.785037 24126 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.785434 24126 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 151 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:16.785805 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 151 candidate_status { last_received { term: 150 index: 118 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:16.785823 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 151 candidate_status { last_received { term: 150 index: 118 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:16.785971 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 151 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 151: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:16.786758 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 151.
I20251215 21:06:16.787063 23792 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b; no voters: bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:16.787305 24126 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 151 FOLLOWER]: Leader election won for term 151
I20251215 21:06:16.787391 24126 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 151 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:16.787539 24126 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 150.118, Last appended by leader: 118, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.789186 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 150 to 151, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 151 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:16.818687 24130 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 151 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:16.818817 24130 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 151 FOLLOWER]: Advancing to term 152
I20251215 21:06:16.819689 24130 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.820070 24130 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 152 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:16.820472 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 152 candidate_status { last_received { term: 150 index: 118 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:16.820472 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 152 candidate_status { last_received { term: 150 index: 118 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:16.820611 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 151 FOLLOWER]: Advancing to term 152
I20251215 21:06:16.821399 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 152.
I20251215 21:06:16.821648 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:16.821820 24130 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 152 FOLLOWER]: Leader election won for term 152
I20251215 21:06:16.821977 24130 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 152 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:16.822118 24130 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 150.118, Last appended by leader: 118, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.823385 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 151 to 152, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 152 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:16.886862 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 152 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 151. Current term is 152. Ops: []
I20251215 21:06:16.887295 24126 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251215 21:06:16.887509 24131 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 151 LEADER]: Stepping down as leader of term 151
I20251215 21:06:16.887568 24131 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 151 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:16.887702 24131 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 151.119, Last appended by leader: 119, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:16.887845 24131 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 151 FOLLOWER]: Advancing to term 152
I20251215 21:06:16.914732 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 152 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 151 index: 119. Preceding OpId from leader: term: 152 index: 119. (term mismatch)
I20251215 21:06:16.914831 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 118
I20251215 21:06:16.914860 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 151.119
I20251215 21:06:16.914887 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 152 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:16.915290 24130 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251215 21:06:16.916286 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 152 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 150 index: 118. Preceding OpId from leader: term: 152 index: 119. (index mismatch)
I20251215 21:06:16.916631 24134 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251215 21:06:17.020013 24124 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 152 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:17.020159 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 152 FOLLOWER]: Advancing to term 153
I20251215 21:06:17.021131 24124 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.021546 24124 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 153 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:17.021965 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 153 candidate_status { last_received { term: 152 index: 119 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:17.021973 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 153 candidate_status { last_received { term: 152 index: 119 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:17.022432 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 153 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:17.022598 24124 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 153 FOLLOWER]: Leader election lost for term 153. Reason: could not achieve majority
I20251215 21:06:17.039124 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 153 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 152. Current term is 153. Ops: []
I20251215 21:06:17.039526 24135 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 152.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251215 21:06:17.039717 24130 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 152 LEADER]: Stepping down as leader of term 152
I20251215 21:06:17.039774 24130 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 152 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:17.039896 24130 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 152.119, Last appended by leader: 119, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.040036 24130 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 152 FOLLOWER]: Advancing to term 153
I20251215 21:06:17.113075 24131 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 152 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:17.113220 24131 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 152 FOLLOWER]: Advancing to term 153
I20251215 21:06:17.114115 24131 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.114493 24131 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 153 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:17.114881 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 153 candidate_status { last_received { term: 152 index: 119 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:17.114885 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 153 candidate_status { last_received { term: 152 index: 119 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:17.115015 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 153 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 153: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:17.115834 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 153.
I20251215 21:06:17.116096 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:17.116253 24131 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 153 FOLLOWER]: Leader election won for term 153
I20251215 21:06:17.116338 24131 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 153 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:17.116493 24131 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 152.119, Last appended by leader: 119, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.117801 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 152 to 153, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 153 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:17.135027 24124 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 153 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:17.135171 24124 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 153 FOLLOWER]: Advancing to term 154
I20251215 21:06:17.136062 24124 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.136472 24124 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 154 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:17.136852 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 154 candidate_status { last_received { term: 152 index: 119 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:17.136858 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 154 candidate_status { last_received { term: 152 index: 119 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:17.136956 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 153 FOLLOWER]: Advancing to term 154
I20251215 21:06:17.137712 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 154.
I20251215 21:06:17.137967 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:17.138162 24124 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 154 FOLLOWER]: Leader election won for term 154
I20251215 21:06:17.138252 24124 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 154 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:17.138383 24124 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 152.119, Last appended by leader: 119, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.139643 23647 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 153 to 154, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 154 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:17.206377 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 154 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 153. Current term is 154. Ops: []
I20251215 21:06:17.206770 24131 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251215 21:06:17.206939 24126 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 153 LEADER]: Stepping down as leader of term 153
I20251215 21:06:17.206992 24126 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 153 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:17.207113 24126 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 153.120, Last appended by leader: 120, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.207248 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 153 FOLLOWER]: Advancing to term 154
I20251215 21:06:17.241762 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 154 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 153 index: 120. Preceding OpId from leader: term: 154 index: 120. (term mismatch)
I20251215 21:06:17.241850 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 119
I20251215 21:06:17.241887 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 153.120
I20251215 21:06:17.241912 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 154 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:17.242332 24124 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251215 21:06:17.244153 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 154 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 152 index: 119. Preceding OpId from leader: term: 154 index: 120. (index mismatch)
I20251215 21:06:17.244480 24136 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251215 21:06:17.295063 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:17.520331 24135 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 154 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:17.520479 24135 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 154 FOLLOWER]: Advancing to term 155
I20251215 21:06:17.521587 24135 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.522125 24135 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 155 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:17.522554 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 155 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:17.522554 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 155 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:17.523056 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef; no voters: 31ff5cfb5d624764ae78ac08450e9022, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:17.523216 24135 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I20251215 21:06:17.526783 24126 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 154 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:17.526886 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 154 FOLLOWER]: Advancing to term 155
I20251215 21:06:17.527762 24126 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.528147 24126 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 155 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:17.528498 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 155 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:17.528499 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 155 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:17.528648 23751 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 155: Already voted for candidate bf42b6838bde4b4199f10f61556786ef in this term.
I20251215 21:06:17.528957 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef
I20251215 21:06:17.529120 24126 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I20251215 21:06:17.530525 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 155 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 154. Current term is 155. Ops: []
I20251215 21:06:17.530840 24136 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 154.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251215 21:06:17.531003 24133 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 154 LEADER]: Stepping down as leader of term 154
I20251215 21:06:17.531055 24133 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:17.531208 24133 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 154.120, Last appended by leader: 120, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.531358 24133 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 154 FOLLOWER]: Advancing to term 155
I20251215 21:06:17.644599 24126 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 155 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:17.644730 24126 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 155 FOLLOWER]: Advancing to term 156
I20251215 21:06:17.645653 24126 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.646047 24126 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 156 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:17.646469 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 156 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:17.646469 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 156 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:17.646623 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 155 FOLLOWER]: Advancing to term 156
I20251215 21:06:17.646623 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 155 FOLLOWER]: Advancing to term 156
I20251215 21:06:17.647583 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 156.
I20251215 21:06:17.647583 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 156.
I20251215 21:06:17.647733 24135 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 156 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:17.647820 24135 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 156 FOLLOWER]: Advancing to term 157
I20251215 21:06:17.647922 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 156 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters:
I20251215 21:06:17.648099 24126 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 156 FOLLOWER]: Leader election won for term 156
I20251215 21:06:17.648196 24126 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 156 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:17.648331 24126 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 154.120, Last appended by leader: 120, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.648855 24135 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.649341 24135 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 157 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:17.649923 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 154 to 156, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 156 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:17.650553 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 157 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:17.650775 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 157 candidate_status { last_received { term: 154 index: 120 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:17.650867 23902 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 156 FOLLOWER]: Advancing to term 157
I20251215 21:06:17.651707 23902 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 157.
I20251215 21:06:17.651978 23716 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:17.652298 24135 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 FOLLOWER]: Leader election won for term 157
I20251215 21:06:17.652390 24135 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:17.652629 24135 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 154.120, Last appended by leader: 120, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.653788 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 156 to 157, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 157 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:17.727978 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 LEADER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 156. Current term is 157. Ops: []
I20251215 21:06:17.728418 24126 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251215 21:06:17.728578 24126 consensus_queue.cc:1243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer bf42b6838bde4b4199f10f61556786ef log is divergent from this leader: its last log entry 157.121 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 120
I20251215 21:06:17.728641 24131 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 156 LEADER]: Stepping down as leader of term 156
I20251215 21:06:17.728711 24131 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 156 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:17.728891 24131 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 156.121, Last appended by leader: 121, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:17.728950 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 157 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 154 index: 120. Preceding OpId from leader: term: 157 index: 121. (index mismatch)
I20251215 21:06:17.729070 24131 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 156 FOLLOWER]: Advancing to term 157
I20251215 21:06:17.729295 24135 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251215 21:06:17.730563 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 157 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 156 index: 121. Preceding OpId from leader: term: 157 index: 121. (term mismatch)
I20251215 21:06:17.730628 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 120
I20251215 21:06:17.730654 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 156.121
I20251215 21:06:17.730718 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 157 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:17.731412 24140 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251215 21:06:18.007982 24133 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 157 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:18.008131 24133 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 157 FOLLOWER]: Advancing to term 158
I20251215 21:06:18.009104 24133 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.009594 24133 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 158 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:18.010010 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 158 candidate_status { last_received { term: 157 index: 121 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:18.010010 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 158 candidate_status { last_received { term: 157 index: 121 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:18.010402 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:18.010555 24133 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20251215 21:06:18.018796 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 157. Current term is 158. Ops: []
I20251215 21:06:18.019173 24135 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 157.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20251215 21:06:18.019353 24130 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 LEADER]: Stepping down as leader of term 157
I20251215 21:06:18.019407 24130 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:18.019518 24130 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 157.121, Last appended by leader: 121, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.019650 24130 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 157 FOLLOWER]: Advancing to term 158
I20251215 21:06:18.032513 24131 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 157 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:18.032639 24131 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 157 FOLLOWER]: Advancing to term 158
I20251215 21:06:18.033540 24131 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.034011 24131 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 158 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:18.034401 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 158 candidate_status { last_received { term: 157 index: 121 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:18.034405 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 158 candidate_status { last_received { term: 157 index: 121 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:18.034543 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 158: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:18.035363 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 158.
I20251215 21:06:18.035655 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:18.035862 24131 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 158 FOLLOWER]: Leader election won for term 158
I20251215 21:06:18.035948 24131 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 158 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:18.036096 24131 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 157.121, Last appended by leader: 121, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.037472 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 157 to 158, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 158 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:18.120687 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 157 index: 121. Preceding OpId from leader: term: 158 index: 122. (index mismatch)
I20251215 21:06:18.121119 24131 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20251215 21:06:18.123034 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 158 FOLLOWER]: Refusing update from remote peer ec6828f7b4fb41348830bc708ea09c2b: Log matching property violated. Preceding OpId in replica: term: 157 index: 121. Preceding OpId from leader: term: 158 index: 122. (index mismatch)
I20251215 21:06:18.123371 24126 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20251215 21:06:18.223317 24133 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Starting leader election (detected failure of leader ec6828f7b4fb41348830bc708ea09c2b)
I20251215 21:06:18.223455 24133 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 158 FOLLOWER]: Advancing to term 159
I20251215 21:06:18.224434 24133 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.225001 24133 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 159 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:18.225338 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 159 candidate_status { last_received { term: 158 index: 122 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:18.225382 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 159 candidate_status { last_received { term: 158 index: 122 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:18.225484 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 158 FOLLOWER]: Advancing to term 159
I20251215 21:06:18.226293 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 159.
I20251215 21:06:18.226555 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:18.226702 24133 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 159 FOLLOWER]: Leader election won for term 159
I20251215 21:06:18.226910 24133 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 159 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:18.227070 24133 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 158.122, Last appended by leader: 122, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.228466 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 158 to 159, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 159 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:18.238734 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 159 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 158. Current term is 159. Ops: []
I20251215 21:06:18.239137 24131 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 158.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251215 21:06:18.239326 24131 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 158 LEADER]: Stepping down as leader of term 158
I20251215 21:06:18.239372 24131 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 158 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:18.239504 24131 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 158.122, Last appended by leader: 122, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.239660 24131 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 158 FOLLOWER]: Advancing to term 159
I20251215 21:06:18.295382 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:18.317718 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 159 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 158 index: 122. Preceding OpId from leader: term: 159 index: 123. (index mismatch)
I20251215 21:06:18.318136 24133 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251215 21:06:18.320122 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 159 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 158 index: 122. Preceding OpId from leader: term: 159 index: 123. (index mismatch)
I20251215 21:06:18.320502 24143 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251215 21:06:18.421756 24135 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 159 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:18.421895 24135 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 159 FOLLOWER]: Advancing to term 160
I20251215 21:06:18.422789 24135 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.423226 24135 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 160 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:18.423655 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 160 candidate_status { last_received { term: 159 index: 123 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:18.423655 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 160 candidate_status { last_received { term: 159 index: 123 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:18.423803 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 159 FOLLOWER]: Advancing to term 160
I20251215 21:06:18.424551 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 160.
I20251215 21:06:18.424823 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 160 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:18.424990 24135 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 160 FOLLOWER]: Leader election won for term 160
I20251215 21:06:18.425076 24135 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 160 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:18.425220 24135 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 159.123, Last appended by leader: 123, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.426539 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 159 to 160, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 160 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:18.429677 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 160 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 159. Current term is 160. Ops: []
I20251215 21:06:18.430003 24142 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 159.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251215 21:06:18.430171 24142 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 159 LEADER]: Stepping down as leader of term 159
I20251215 21:06:18.430222 24142 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 159 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:18.430362 24142 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 159.123, Last appended by leader: 123, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.430513 24142 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 159 FOLLOWER]: Advancing to term 160
I20251215 21:06:18.521840 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 160 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 159 index: 123. Preceding OpId from leader: term: 160 index: 124. (index mismatch)
I20251215 21:06:18.522248 24135 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251215 21:06:18.524258 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 160 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 159 index: 123. Preceding OpId from leader: term: 160 index: 124. (index mismatch)
I20251215 21:06:18.524647 24135 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251215 21:06:18.806991 24142 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 160 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:18.807122 24142 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 160 FOLLOWER]: Advancing to term 161
I20251215 21:06:18.808032 24142 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.808435 24142 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 161 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:18.808877 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 161 candidate_status { last_received { term: 160 index: 124 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:18.808877 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 161 candidate_status { last_received { term: 160 index: 124 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:18.809273 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 161 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:18.809424 24142 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 161 FOLLOWER]: Leader election lost for term 161. Reason: could not achieve majority
I20251215 21:06:18.823841 23902 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 161 FOLLOWER]: Rejecting Update request from peer bf42b6838bde4b4199f10f61556786ef for earlier term 160. Current term is 161. Ops: []
I20251215 21:06:18.824257 24145 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: INVALID_TERM, Last received: 160.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251215 21:06:18.824460 24135 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 160 LEADER]: Stepping down as leader of term 160
I20251215 21:06:18.824517 24135 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 160 LEADER]: Becoming Follower/Learner. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:18.824656 24135 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 160.124, Last appended by leader: 124, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.824846 24135 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 160 FOLLOWER]: Advancing to term 161
I20251215 21:06:18.886734 24131 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 160 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:18.886866 24131 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 160 FOLLOWER]: Advancing to term 161
I20251215 21:06:18.887844 24131 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.888264 24131 leader_election.cc:290] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 161 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:18.888662 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 161 candidate_status { last_received { term: 160 index: 124 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:18.888662 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "ec6828f7b4fb41348830bc708ea09c2b" candidate_term: 161 candidate_status { last_received { term: 160 index: 124 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:18.888855 23902 raft_consensus.cc:2393] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 161 FOLLOWER]: Leader election vote request: Denying vote to candidate ec6828f7b4fb41348830bc708ea09c2b in current term 161: Already voted for candidate 31ff5cfb5d624764ae78ac08450e9022 in this term.
I20251215 21:06:18.889621 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec6828f7b4fb41348830bc708ea09c2b in term 161.
I20251215 21:06:18.889865 23791 leader_election.cc:304] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:18.890017 24131 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 161 FOLLOWER]: Leader election won for term 161
I20251215 21:06:18.890165 24131 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 161 LEADER]: Becoming Leader. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:18.890302 24131 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 160.124, Last appended by leader: 124, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.891572 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b reported cstate change: term changed from 160 to 161, leader changed from bf42b6838bde4b4199f10f61556786ef (127.23.16.65) to ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66). New cstate: current_term: 161 leader_uuid: "ec6828f7b4fb41348830bc708ea09c2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:18.931913 24142 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 161 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:18.932054 24142 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 161 FOLLOWER]: Advancing to term 162
I20251215 21:06:18.932998 24142 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.933409 24142 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 162 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
I20251215 21:06:18.933794 23751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 162 candidate_status { last_received { term: 160 index: 124 } } ignore_live_leader: false dest_uuid: "bf42b6838bde4b4199f10f61556786ef"
I20251215 21:06:18.933794 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "31ff5cfb5d624764ae78ac08450e9022" candidate_term: 162 candidate_status { last_received { term: 160 index: 124 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:18.933923 23751 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 161 FOLLOWER]: Advancing to term 162
I20251215 21:06:18.934690 23751 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 162 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ff5cfb5d624764ae78ac08450e9022 in term 162.
I20251215 21:06:18.934947 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 162 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022, bf42b6838bde4b4199f10f61556786ef; no voters: ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:18.935101 24142 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 162 FOLLOWER]: Leader election won for term 162
I20251215 21:06:18.935245 24142 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 162 LEADER]: Becoming Leader. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:18.935379 24142 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 160.124, Last appended by leader: 124, Current term: 162, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.936686 23646 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 reported cstate change: term changed from 161 to 162, leader changed from ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66) to 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67). New cstate: current_term: 162 leader_uuid: "31ff5cfb5d624764ae78ac08450e9022" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:18.978266 23751 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 162 FOLLOWER]: Rejecting Update request from peer ec6828f7b4fb41348830bc708ea09c2b for earlier term 161. Current term is 162. Ops: []
I20251215 21:06:18.978662 24131 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251215 21:06:18.978854 24146 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 161 LEADER]: Stepping down as leader of term 161
I20251215 21:06:18.978926 24146 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 161 LEADER]: Becoming Follower/Learner. State: Replica: ec6828f7b4fb41348830bc708ea09c2b, State: Running, Role: LEADER
I20251215 21:06:18.979086 24146 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 161.125, Last appended by leader: 125, Current term: 161, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:18.979255 24146 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 161 FOLLOWER]: Advancing to term 162
I20251215 21:06:19.013978 23751 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 162 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 160 index: 124. Preceding OpId from leader: term: 162 index: 125. (index mismatch)
I20251215 21:06:19.014389 24142 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251215 21:06:19.016355 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 162 FOLLOWER]: Refusing update from remote peer 31ff5cfb5d624764ae78ac08450e9022: Log matching property violated. Preceding OpId in replica: term: 161 index: 125. Preceding OpId from leader: term: 162 index: 125. (term mismatch)
I20251215 21:06:19.016418 23826 pending_rounds.cc:85] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting all ops after (but not including) 124
I20251215 21:06:19.016443 23826 pending_rounds.cc:107] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: Aborting uncommitted NO_OP operation due to leader change: 161.125
I20251215 21:06:19.016466 23826 raft_consensus.cc:2889] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 162 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:19.016834 24148 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251215 21:06:19.117084 24135 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 162 FOLLOWER]: Starting leader election (detected failure of leader 31ff5cfb5d624764ae78ac08450e9022)
I20251215 21:06:19.117230 24135 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 162 FOLLOWER]: Advancing to term 163
I20251215 21:06:19.118144 24135 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:19.118549 24135 leader_election.cc:290] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 163 election: Requested vote from peers 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67:41817), ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057)
I20251215 21:06:19.118943 23826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 163 candidate_status { last_received { term: 162 index: 125 } } ignore_live_leader: false dest_uuid: "ec6828f7b4fb41348830bc708ea09c2b"
I20251215 21:06:19.118940 23902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fee89533731c48b8979083516919521b" candidate_uuid: "bf42b6838bde4b4199f10f61556786ef" candidate_term: 163 candidate_status { last_received { term: 162 index: 125 } } ignore_live_leader: false dest_uuid: "31ff5cfb5d624764ae78ac08450e9022"
I20251215 21:06:19.119079 23826 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 162 FOLLOWER]: Advancing to term 163
I20251215 21:06:19.119853 23826 raft_consensus.cc:2468] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 163 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf42b6838bde4b4199f10f61556786ef in term 163.
I20251215 21:06:19.120105 23717 leader_election.cc:304] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [CANDIDATE]: Term 163 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b; no voters: 31ff5cfb5d624764ae78ac08450e9022
I20251215 21:06:19.120308 24135 raft_consensus.cc:2804] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 163 FOLLOWER]: Leader election won for term 163
I20251215 21:06:19.120402 24135 raft_consensus.cc:697] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 163 LEADER]: Becoming Leader. State: Replica: bf42b6838bde4b4199f10f61556786ef, State: Running, Role: LEADER
I20251215 21:06:19.120551 24135 consensus_queue.cc:237] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 162.125, Last appended by leader: 125, Current term: 163, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:19.121924 23644 catalog_manager.cc:5654] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef reported cstate change: term changed from 162 to 163, leader changed from 31ff5cfb5d624764ae78ac08450e9022 (127.23.16.67) to bf42b6838bde4b4199f10f61556786ef (127.23.16.65). New cstate: current_term: 163 leader_uuid: "bf42b6838bde4b4199f10f61556786ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:19.127110 23826 raft_consensus.cc:1240] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 163 FOLLOWER]: Rejecting Update request from peer 31ff5cfb5d624764ae78ac08450e9022 for earlier term 162. Current term is 163. Ops: []
I20251215 21:06:19.127480 24142 consensus_queue.cc:1059] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: INVALID_TERM, Last received: 162.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251215 21:06:19.127624 24142 raft_consensus.cc:3055] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 162 LEADER]: Stepping down as leader of term 162
I20251215 21:06:19.127671 24142 raft_consensus.cc:740] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 162 LEADER]: Becoming Follower/Learner. State: Replica: 31ff5cfb5d624764ae78ac08450e9022, State: Running, Role: LEADER
I20251215 21:06:19.127800 24142 consensus_queue.cc:260] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 162.125, Last appended by leader: 125, Current term: 162, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:19.127971 24142 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 162 FOLLOWER]: Advancing to term 163
I20251215 21:06:19.202986 23902 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 163 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 162 index: 125. Preceding OpId from leader: term: 163 index: 126. (index mismatch)
I20251215 21:06:19.203390 24135 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251215 21:06:19.205276 23826 raft_consensus.cc:1275] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 163 FOLLOWER]: Refusing update from remote peer bf42b6838bde4b4199f10f61556786ef: Log matching property violated. Preceding OpId in replica: term: 162 index: 125. Preceding OpId from leader: term: 163 index: 126. (index mismatch)
I20251215 21:06:19.205639 24149 consensus_queue.cc:1048] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251215 21:06:19.295671 23617 timestamp_advancement-itest.cc:267] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:267: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:332: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20251215 21:06:19.296087 23617 tablet_server.cc:178] TabletServer@127.23.16.65:0 shutting down...
I20251215 21:06:19.309934 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:19.310197 23617 tablet_replica.cc:333] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef: stopping tablet replica
I20251215 21:06:19.310400 23617 raft_consensus.cc:2243] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 163 LEADER]: Raft consensus shutting down.
I20251215 21:06:19.310611 23617 raft_consensus.cc:2272] T fee89533731c48b8979083516919521b P bf42b6838bde4b4199f10f61556786ef [term 163 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:19.314039 23617 tablet_server.cc:195] TabletServer@127.23.16.65:0 shutdown complete.
I20251215 21:06:19.317395 23617 tablet_server.cc:178] TabletServer@127.23.16.66:0 shutting down...
I20251215 21:06:19.348105 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:19.348318 23617 tablet_replica.cc:333] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b: stopping tablet replica
I20251215 21:06:19.348428 23617 raft_consensus.cc:2243] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 163 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:19.348546 23617 raft_consensus.cc:2272] T fee89533731c48b8979083516919521b P ec6828f7b4fb41348830bc708ea09c2b [term 163 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:19.353127 24142 raft_consensus.cc:493] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 163 FOLLOWER]: Starting leader election (detected failure of leader bf42b6838bde4b4199f10f61556786ef)
I20251215 21:06:19.353209 24142 raft_consensus.cc:3060] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 163 FOLLOWER]: Advancing to term 164
I20251215 21:06:19.353976 24142 raft_consensus.cc:515] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ff5cfb5d624764ae78ac08450e9022" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 41817 } } peers { permanent_uuid: "ec6828f7b4fb41348830bc708ea09c2b" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 44057 } } peers { permanent_uuid: "bf42b6838bde4b4199f10f61556786ef" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 46839 } }
I20251215 21:06:19.354311 24142 leader_election.cc:290] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 164 election: Requested vote from peers ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057), bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839)
W20251215 21:06:19.354786 23866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.16.65:46839: connect: Connection refused (error 111)
W20251215 21:06:19.355360 23868 leader_election.cc:336] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 164 election: RPC error from VoteRequest() call to peer ec6828f7b4fb41348830bc708ea09c2b (127.23.16.66:44057): Network error: Client connection negotiation failed: client connection to 127.23.16.66:44057: connect: Connection refused (error 111)
W20251215 21:06:19.355439 23866 leader_election.cc:336] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 164 election: RPC error from VoteRequest() call to peer bf42b6838bde4b4199f10f61556786ef (127.23.16.65:46839): Network error: Client connection negotiation failed: client connection to 127.23.16.65:46839: connect: Connection refused (error 111)
I20251215 21:06:19.355510 23866 leader_election.cc:304] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [CANDIDATE]: Term 164 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ff5cfb5d624764ae78ac08450e9022; no voters: bf42b6838bde4b4199f10f61556786ef, ec6828f7b4fb41348830bc708ea09c2b
I20251215 21:06:19.355625 24142 raft_consensus.cc:2749] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 164 FOLLOWER]: Leader election lost for term 164. Reason: could not achieve majority
I20251215 21:06:19.360944 23617 tablet_server.cc:195] TabletServer@127.23.16.66:0 shutdown complete.
I20251215 21:06:19.362530 23617 tablet_server.cc:178] TabletServer@127.23.16.67:0 shutting down...
I20251215 21:06:19.380466 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:19.380656 23617 tablet_replica.cc:333] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022: stopping tablet replica
I20251215 21:06:19.380759 23617 raft_consensus.cc:2243] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 164 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:19.380897 23617 raft_consensus.cc:2272] T fee89533731c48b8979083516919521b P 31ff5cfb5d624764ae78ac08450e9022 [term 164 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:19.393435 23617 tablet_server.cc:195] TabletServer@127.23.16.67:0 shutdown complete.
I20251215 21:06:19.395247 23617 master.cc:561] Master@127.23.16.126:42903 shutting down...
I20251215 21:06:19.479333 23617 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:06:19.479528 23617 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:19.479604 23617 tablet_replica.cc:333] T 00000000000000000000000000000000 P 656f090a71e9497b9a25585b2f7091a8: stopping tablet replica
I20251215 21:06:19.492321 23617 master.cc:583] Master@127.23.16.126:42903 shutdown complete.
I20251215 21:06:19.496311 23617 test_util.cc:183] -----------------------------------------------
I20251215 21:06:19.496362 23617 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765832746896456-23617-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32590 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20251215 21:06:19.496986 23617 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.16.126:44931
I20251215 21:06:19.497301 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:19.498673 24153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:19.498778 24154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.498930 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:19.498792 24156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.499128 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:19.499168 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:19.499182 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832779499182 us; error 0 us; skew 500 ppm
I20251215 21:06:19.499913 23617 webserver.cc:492] Webserver started at http://127.23.16.126:41515/ using document root <none> and password file <none>
I20251215 21:06:19.500042 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:19.500082 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:19.500140 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:19.500464 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/master-0-root/instance:
uuid: "a0af3555a77c41f7bc301b5e61c48f5e"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.501817 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251215 21:06:19.502599 24161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.502803 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:06:19.502864 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/master-0-root
uuid: "a0af3555a77c41f7bc301b5e61c48f5e"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.502930 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:19.512694 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:19.513077 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:19.516876 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.126:44931
I20251215 21:06:19.518239 24213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.126:44931 every 8 connection(s)
I20251215 21:06:19.518648 24214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:19.520303 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e: Bootstrap starting.
I20251215 21:06:19.521034 24214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:19.521860 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e: No bootstrap required, opened a new log
I20251215 21:06:19.522189 24214 raft_consensus.cc:359] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" member_type: VOTER }
I20251215 21:06:19.522269 24214 raft_consensus.cc:385] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:19.522295 24214 raft_consensus.cc:740] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0af3555a77c41f7bc301b5e61c48f5e, State: Initialized, Role: FOLLOWER
I20251215 21:06:19.522385 24214 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" member_type: VOTER }
I20251215 21:06:19.522437 24214 raft_consensus.cc:399] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:06:19.522462 24214 raft_consensus.cc:493] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:06:19.522495 24214 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:19.523093 24214 raft_consensus.cc:515] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" member_type: VOTER }
I20251215 21:06:19.523195 24214 leader_election.cc:304] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0af3555a77c41f7bc301b5e61c48f5e; no voters:
I20251215 21:06:19.523330 24214 leader_election.cc:290] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:06:19.523455 24217 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:06:19.523651 24217 raft_consensus.cc:697] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 1 LEADER]: Becoming Leader. State: Replica: a0af3555a77c41f7bc301b5e61c48f5e, State: Running, Role: LEADER
I20251215 21:06:19.523663 24214 sys_catalog.cc:565] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:06:19.523810 24217 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" member_type: VOTER }
I20251215 21:06:19.524211 24218 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" member_type: VOTER } }
I20251215 21:06:19.524317 24218 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:19.524591 24224 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:06:19.524223 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0af3555a77c41f7bc301b5e61c48f5e. Latest consensus state: current_term: 1 leader_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af3555a77c41f7bc301b5e61c48f5e" member_type: VOTER } }
I20251215 21:06:19.524950 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:19.525846 24224 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:06:19.526055 23617 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 21:06:19.527458 24224 catalog_manager.cc:1357] Generated new cluster ID: a7f7b2236bf0467687b4a3e49f5714dc
I20251215 21:06:19.527503 24224 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:06:19.542899 24224 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:06:19.543421 24224 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:06:19.549209 24224 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e: Generated new TSK 0
I20251215 21:06:19.549347 24224 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:06:19.558311 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:19.560099 24236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:19.560114 24235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.560245 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:19.560142 24238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.560438 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:19.560479 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:19.560493 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832779560493 us; error 0 us; skew 500 ppm
I20251215 21:06:19.561298 23617 webserver.cc:492] Webserver started at http://127.23.16.65:39799/ using document root <none> and password file <none>
I20251215 21:06:19.561437 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:19.561482 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:19.561548 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:19.561913 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root/instance:
uuid: "290befd6e2fa43ad9a5b326619654cfc"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.563235 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.564080 24243 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.564283 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.564339 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "290befd6e2fa43ad9a5b326619654cfc"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.564400 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:19.578536 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:19.578884 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:19.579138 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:19.579573 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:19.579609 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.579651 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:19.579680 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.583461 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:44859
I20251215 21:06:19.583490 24306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:44859 every 8 connection(s)
I20251215 21:06:19.584201 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:19.585739 24311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:19.585850 24314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:19.585966 24312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.586380 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:19.586573 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:19.586611 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:19.586624 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832779586624 us; error 0 us; skew 500 ppm
I20251215 21:06:19.587435 23617 webserver.cc:492] Webserver started at http://127.23.16.66:44163/ using document root <none> and password file <none>
I20251215 21:06:19.587566 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:19.587606 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:19.587657 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:19.587989 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-1-root/instance:
uuid: "a1a30b1f25d743439f4b73d5389c1017"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.589304 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251215 21:06:19.590085 24319 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.590337 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.590400 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-1-root
uuid: "a1a30b1f25d743439f4b73d5389c1017"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.590464 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:19.591372 24307 heartbeater.cc:344] Connected to a master server at 127.23.16.126:44931
I20251215 21:06:19.591454 24307 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:19.591617 24307 heartbeater.cc:507] Master 127.23.16.126:44931 requested a full tablet report, sending...
I20251215 21:06:19.592152 24178 ts_manager.cc:194] Registered new tserver with Master: 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:19.592842 24178 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48142
I20251215 21:06:19.601950 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:19.602254 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:19.602501 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:19.602881 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:19.602918 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.602957 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:19.602983 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.606945 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.66:39861
I20251215 21:06:19.607007 24382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.66:39861 every 8 connection(s)
I20251215 21:06:19.607693 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:19.609274 24387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:19.609308 24386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.609534 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:19.609827 24389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:19.610066 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:19.610118 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:19.610138 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832779610138 us; error 0 us; skew 500 ppm
I20251215 21:06:19.610997 23617 webserver.cc:492] Webserver started at http://127.23.16.67:34477/ using document root <none> and password file <none>
I20251215 21:06:19.611193 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:19.611243 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:19.611295 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:19.611435 24383 heartbeater.cc:344] Connected to a master server at 127.23.16.126:44931
I20251215 21:06:19.611517 24383 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:19.611688 24383 heartbeater.cc:507] Master 127.23.16.126:44931 requested a full tablet report, sending...
I20251215 21:06:19.611675 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-2-root/instance:
uuid: "22b7ab1cf11d4d78ba5425555355b386"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.612260 24178 ts_manager.cc:194] Registered new tserver with Master: a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:19.612922 24178 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48154
I20251215 21:06:19.613348 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.614176 24394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.614372 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.614434 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-2-root
uuid: "22b7ab1cf11d4d78ba5425555355b386"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:19.614495 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:19.620442 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:19.620721 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:19.620992 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:19.621356 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:19.621390 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.621428 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:19.621454 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:19.625546 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.67:33457
I20251215 21:06:19.625590 24457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.67:33457 every 8 connection(s)
I20251215 21:06:19.629251 24458 heartbeater.cc:344] Connected to a master server at 127.23.16.126:44931
I20251215 21:06:19.629325 24458 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:19.629492 24458 heartbeater.cc:507] Master 127.23.16.126:44931 requested a full tablet report, sending...
I20251215 21:06:19.630030 24178 ts_manager.cc:194] Registered new tserver with Master: 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457)
I20251215 21:06:19.630323 23617 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00442841s
I20251215 21:06:19.630553 24178 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48158
I20251215 21:06:19.639381 23617 test_util.cc:276] Using random seed: -74438061
I20251215 21:06:19.645267 24178 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48178:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:06:19.645640 24178 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:06:19.653463 24422 tablet_service.cc:1505] Processing CreateTablet for tablet 3c998c8faa354807b3ef99adc3b97252 (DEFAULT_TABLE table=test-workload [id=5ac63f32d8924080b4356c93510a55ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:19.653761 24422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c998c8faa354807b3ef99adc3b97252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:19.654415 24271 tablet_service.cc:1505] Processing CreateTablet for tablet 3c998c8faa354807b3ef99adc3b97252 (DEFAULT_TABLE table=test-workload [id=5ac63f32d8924080b4356c93510a55ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:19.654600 24271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c998c8faa354807b3ef99adc3b97252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:19.655812 24479 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386: Bootstrap starting.
I20251215 21:06:19.656446 24480 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Bootstrap starting.
I20251215 21:06:19.656452 24347 tablet_service.cc:1505] Processing CreateTablet for tablet 3c998c8faa354807b3ef99adc3b97252 (DEFAULT_TABLE table=test-workload [id=5ac63f32d8924080b4356c93510a55ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:19.656702 24347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c998c8faa354807b3ef99adc3b97252. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:19.656824 24479 tablet_bootstrap.cc:654] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:19.657423 24480 tablet_bootstrap.cc:654] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:19.657711 24479 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386: No bootstrap required, opened a new log
I20251215 21:06:19.657788 24479 ts_tablet_manager.cc:1403] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:06:19.658211 24479 raft_consensus.cc:359] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.658412 24480 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: No bootstrap required, opened a new log
I20251215 21:06:19.658524 24479 raft_consensus.cc:385] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:19.658596 24480 ts_tablet_manager.cc:1403] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:06:19.658625 24479 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b7ab1cf11d4d78ba5425555355b386, State: Initialized, Role: FOLLOWER
I20251215 21:06:19.658836 24479 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.659037 24480 raft_consensus.cc:359] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.659125 24480 raft_consensus.cc:385] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:19.659183 24458 heartbeater.cc:499] Master 127.23.16.126:44931 was elected leader, sending a full tablet report...
I20251215 21:06:19.659195 24480 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290befd6e2fa43ad9a5b326619654cfc, State: Initialized, Role: FOLLOWER
I20251215 21:06:19.659186 24479 ts_tablet_manager.cc:1434] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.659382 24480 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.659432 24483 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017: Bootstrap starting.
I20251215 21:06:19.659698 24480 ts_tablet_manager.cc:1434] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.659880 24307 heartbeater.cc:499] Master 127.23.16.126:44931 was elected leader, sending a full tablet report...
I20251215 21:06:19.660365 24483 tablet_bootstrap.cc:654] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:19.661465 24483 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017: No bootstrap required, opened a new log
I20251215 21:06:19.661545 24483 ts_tablet_manager.cc:1403] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:06:19.661900 24483 raft_consensus.cc:359] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.661967 24483 raft_consensus.cc:385] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:19.661986 24483 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Initialized, Role: FOLLOWER
I20251215 21:06:19.662075 24483 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.662369 24383 heartbeater.cc:499] Master 127.23.16.126:44931 was elected leader, sending a full tablet report...
I20251215 21:06:19.662353 24483 ts_tablet_manager.cc:1434] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:19.686367 24487 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:06:19.686509 24487 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.687007 24487 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:19.688958 24484 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:06:19.689077 24484 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.689610 24484 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:19.691017 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386" is_pre_election: true
I20251215 21:06:19.691140 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc" is_pre_election: true
I20251215 21:06:19.691155 24432 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a30b1f25d743439f4b73d5389c1017 in term 0.
I20251215 21:06:19.691233 24281 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a30b1f25d743439f4b73d5389c1017 in term 0.
I20251215 21:06:19.691566 24321 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017; no voters:
I20251215 21:06:19.691802 24487 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:06:19.691881 24487 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:19.691926 24487 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:19.693032 24487 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.693627 24487 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 1 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:19.693785 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:19.693863 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:19.693897 24432 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:19.693902 24280 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc" is_pre_election: true
I20251215 21:06:19.693960 24281 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:19.694954 24281 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a30b1f25d743439f4b73d5389c1017 in term 1.
I20251215 21:06:19.694954 24432 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a30b1f25d743439f4b73d5389c1017 in term 1.
I20251215 21:06:19.695258 24323 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017; no voters:
I20251215 21:06:19.695302 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017" is_pre_election: true
I20251215 21:06:19.695407 24357 raft_consensus.cc:2393] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22b7ab1cf11d4d78ba5425555355b386 in current term 1: Already voted for candidate a1a30b1f25d743439f4b73d5389c1017 in this term.
I20251215 21:06:19.695484 24487 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:06:19.695672 24396 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386; no voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:19.695693 24487 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 LEADER]: Becoming Leader. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Running, Role: LEADER
I20251215 21:06:19.695802 24484 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251215 21:06:19.695987 24487 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:19.697356 24178 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66). New cstate: current_term: 1 leader_uuid: "a1a30b1f25d743439f4b73d5389c1017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:19.719002 24281 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 1 FOLLOWER]: Refusing update from remote peer a1a30b1f25d743439f4b73d5389c1017: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:06:19.719056 24432 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 1 FOLLOWER]: Refusing update from remote peer a1a30b1f25d743439f4b73d5389c1017: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:06:19.719575 24487 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:06:19.719977 24491 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:06:19.723171 24498 mvcc.cc:204] Tried to move back new op lower bound from 7232851065730301952 to 7232851065636016128. Current Snapshot: MvccSnapshot[applied={T|T < 7232851065727467520}]
I20251215 21:06:20.033170 24484 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a1a30b1f25d743439f4b73d5389c1017)
I20251215 21:06:20.033300 24484 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:06:20.034217 24484 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.034605 24484 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 2 election: Requested vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.034989 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.034992 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.035362 24398 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386; no voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:20.035529 24484 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 21:06:20.056562 24432 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 2 FOLLOWER]: Rejecting Update request from peer a1a30b1f25d743439f4b73d5389c1017 for earlier term 1. Current term is 2. Ops: []
I20251215 21:06:20.056982 24516 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: INVALID_TERM, Last received: 1.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20251215 21:06:20.086856 24485 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a1a30b1f25d743439f4b73d5389c1017)
I20251215 21:06:20.087003 24485 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:06:20.087988 24485 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.088490 24485 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 2 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.088981 24281 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 2 FOLLOWER]: Rejecting Update request from peer a1a30b1f25d743439f4b73d5389c1017 for earlier term 1. Current term is 2. Ops: []
I20251215 21:06:20.089444 24516 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: INVALID_TERM, Last received: 1.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20251215 21:06:20.091933 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:20.092060 24432 raft_consensus.cc:2393] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 290befd6e2fa43ad9a5b326619654cfc in current term 2: Already voted for candidate 22b7ab1cf11d4d78ba5425555355b386 in this term.
I20251215 21:06:20.092329 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.092572 24245 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc; no voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:20.092741 24485 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 21:06:20.100275 24515 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 LEADER]: Stepping down as leader of term 1
I20251215 21:06:20.100347 24515 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Running, Role: LEADER
I20251215 21:06:20.100497 24515 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.100673 24515 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:06:20.136992 24484 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:20.137120 24484 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:06:20.138001 24484 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.138396 24484 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 3 election: Requested vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.138803 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.138806 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.138921 24357 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:06:20.138949 24281 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:06:20.139750 24357 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b7ab1cf11d4d78ba5425555355b386 in term 3.
I20251215 21:06:20.139752 24281 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b7ab1cf11d4d78ba5425555355b386 in term 3.
I20251215 21:06:20.140066 24398 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc; no voters:
I20251215 21:06:20.140236 24484 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 21:06:20.140399 24484 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 3 LEADER]: Becoming Leader. State: Replica: 22b7ab1cf11d4d78ba5425555355b386, State: Running, Role: LEADER
I20251215 21:06:20.140547 24484 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.141933 24178 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 reported cstate change: term changed from 1 to 3, leader changed from a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66) to 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67). New cstate: current_term: 3 leader_uuid: "22b7ab1cf11d4d78ba5425555355b386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:20.232354 24281 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 3 FOLLOWER]: Refusing update from remote peer 22b7ab1cf11d4d78ba5425555355b386: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20251215 21:06:20.232712 24484 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20251215 21:06:20.241416 24357 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 3 FOLLOWER]: Refusing update from remote peer 22b7ab1cf11d4d78ba5425555355b386: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20251215 21:06:20.241771 24484 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20251215 21:06:20.343396 24515 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22b7ab1cf11d4d78ba5425555355b386)
I20251215 21:06:20.343538 24515 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 3 FOLLOWER]: Advancing to term 4
I20251215 21:06:20.344486 24515 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.345062 24515 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 4 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:20.345474 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:20.345489 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.345840 24323 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1a30b1f25d743439f4b73d5389c1017; no voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc
I20251215 21:06:20.346016 24515 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251215 21:06:20.346287 24357 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 4 FOLLOWER]: Rejecting Update request from peer 22b7ab1cf11d4d78ba5425555355b386 for earlier term 3. Current term is 4. Ops: []
I20251215 21:06:20.346648 24521 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: INVALID_TERM, Last received: 3.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251215 21:06:20.407105 24521 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 3 LEADER]: Stepping down as leader of term 3
I20251215 21:06:20.407207 24521 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22b7ab1cf11d4d78ba5425555355b386, State: Running, Role: LEADER
I20251215 21:06:20.407343 24521 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.407508 24521 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 3 FOLLOWER]: Advancing to term 4
I20251215 21:06:20.460544 24515 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:20.460675 24515 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 4 FOLLOWER]: Advancing to term 5
I20251215 21:06:20.461575 24515 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.461992 24515 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 5 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:20.462452 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 5 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.462459 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 5 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:20.462607 24432 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 4 FOLLOWER]: Advancing to term 5
I20251215 21:06:20.463452 24432 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a30b1f25d743439f4b73d5389c1017 in term 5.
I20251215 21:06:20.463742 24321 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017; no voters: 290befd6e2fa43ad9a5b326619654cfc
I20251215 21:06:20.463946 24515 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 5 FOLLOWER]: Leader election won for term 5
I20251215 21:06:20.464040 24515 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 5 LEADER]: Becoming Leader. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Running, Role: LEADER
I20251215 21:06:20.464190 24515 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.465600 24178 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 reported cstate change: term changed from 3 to 5, leader changed from 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67) to a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66). New cstate: current_term: 5 leader_uuid: "a1a30b1f25d743439f4b73d5389c1017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:20.505036 24485 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 3 FOLLOWER]: Starting leader election (detected failure of leader 22b7ab1cf11d4d78ba5425555355b386)
I20251215 21:06:20.505167 24485 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 3 FOLLOWER]: Advancing to term 4
I20251215 21:06:20.506059 24485 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.506500 24485 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 4 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.506904 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:20.506937 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.507018 24432 raft_consensus.cc:2368] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 290befd6e2fa43ad9a5b326619654cfc for earlier term 4. Current term is 5.
I20251215 21:06:20.507334 24245 leader_election.cc:400] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 4 election: Vote denied by peer 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457) with higher term. Message: Invalid argument: T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 290befd6e2fa43ad9a5b326619654cfc for earlier term 4. Current term is 5.
I20251215 21:06:20.507375 24245 leader_election.cc:403] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251215 21:06:20.507493 24485 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 4 FOLLOWER]: Advancing to term 5
I20251215 21:06:20.508203 24485 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457) with higher term. Message: Invalid argument: T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 290befd6e2fa43ad9a5b326619654cfc for earlier term 4. Current term is 5.
I20251215 21:06:20.545691 24281 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 5 FOLLOWER]: Refusing update from remote peer a1a30b1f25d743439f4b73d5389c1017: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20251215 21:06:20.546094 24515 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251215 21:06:20.557878 24432 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Refusing update from remote peer a1a30b1f25d743439f4b73d5389c1017: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20251215 21:06:20.558269 24516 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251215 21:06:20.660179 24521 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a1a30b1f25d743439f4b73d5389c1017)
I20251215 21:06:20.660310 24521 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 5 FOLLOWER]: Advancing to term 6
I20251215 21:06:20.662606 24521 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.663100 24521 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 6 election: Requested vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.663475 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.663475 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.663918 24396 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386; no voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:20.664085 24521 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251215 21:06:20.677093 24432 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 6 FOLLOWER]: Rejecting Update request from peer a1a30b1f25d743439f4b73d5389c1017 for earlier term 5. Current term is 6. Ops: []
I20251215 21:06:20.677471 24491 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: INVALID_TERM, Last received: 5.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251215 21:06:20.726775 24491 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 5 LEADER]: Stepping down as leader of term 5
I20251215 21:06:20.726891 24491 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Running, Role: LEADER
I20251215 21:06:20.727041 24491 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.727219 24491 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 5 FOLLOWER]: Advancing to term 6
I20251215 21:06:20.736905 24485 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 5 FOLLOWER]: Starting leader election (detected failure of leader a1a30b1f25d743439f4b73d5389c1017)
I20251215 21:06:20.737015 24485 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 5 FOLLOWER]: Advancing to term 6
I20251215 21:06:20.737824 24485 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.738204 24485 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 6 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.738627 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.738627 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:20.738806 24432 raft_consensus.cc:2393] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 290befd6e2fa43ad9a5b326619654cfc in current term 6: Already voted for candidate 22b7ab1cf11d4d78ba5425555355b386 in this term.
I20251215 21:06:20.739598 24357 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290befd6e2fa43ad9a5b326619654cfc in term 6.
I20251215 21:06:20.739850 24245 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017; no voters: 22b7ab1cf11d4d78ba5425555355b386
I20251215 21:06:20.740010 24485 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 6 FOLLOWER]: Leader election won for term 6
I20251215 21:06:20.740164 24485 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 6 LEADER]: Becoming Leader. State: Replica: 290befd6e2fa43ad9a5b326619654cfc, State: Running, Role: LEADER
I20251215 21:06:20.740304 24485 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.741652 24177 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc reported cstate change: term changed from 5 to 6, leader changed from a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66) to 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65). New cstate: current_term: 6 leader_uuid: "290befd6e2fa43ad9a5b326619654cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:20.765264 24521 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:20.765383 24521 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 6 FOLLOWER]: Advancing to term 7
I20251215 21:06:20.766216 24521 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.766592 24521 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 7 election: Requested vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:20.766976 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 7 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.766994 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 7 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:20.767122 24357 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 6 FOLLOWER]: Advancing to term 7
I20251215 21:06:20.767798 24357 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b7ab1cf11d4d78ba5425555355b386 in term 7.
I20251215 21:06:20.768069 24396 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017; no voters: 290befd6e2fa43ad9a5b326619654cfc
I20251215 21:06:20.768244 24521 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 FOLLOWER]: Leader election won for term 7
I20251215 21:06:20.768323 24521 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 LEADER]: Becoming Leader. State: Replica: 22b7ab1cf11d4d78ba5425555355b386, State: Running, Role: LEADER
I20251215 21:06:20.768451 24521 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.825815 24175 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 reported cstate change: term changed from 6 to 7, leader changed from 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65) to 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67). New cstate: current_term: 7 leader_uuid: "22b7ab1cf11d4d78ba5425555355b386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:20.841641 24357 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 7 FOLLOWER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 6. Current term is 7. Ops: []
I20251215 21:06:20.841641 24432 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 LEADER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 6. Current term is 7. Ops: []
I20251215 21:06:20.842103 24522 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251215 21:06:20.842370 24485 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251215 21:06:20.842475 24485 consensus_queue.cc:1243] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer 22b7ab1cf11d4d78ba5425555355b386 log is divergent from this leader: its last log entry 7.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20251215 21:06:20.855825 24281 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 6 LEADER]: Stepping down as leader of term 6
I20251215 21:06:20.855911 24281 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 290befd6e2fa43ad9a5b326619654cfc, State: Running, Role: LEADER
I20251215 21:06:20.856020 24281 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 6.19, Last appended by leader: 19, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.856173 24281 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 6 FOLLOWER]: Advancing to term 7
I20251215 21:06:20.857055 24281 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 7 FOLLOWER]: Refusing update from remote peer 22b7ab1cf11d4d78ba5425555355b386: Log matching property violated. Preceding OpId in replica: term: 6 index: 19. Preceding OpId from leader: term: 7 index: 19. (term mismatch)
I20251215 21:06:20.857105 24281 pending_rounds.cc:85] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Aborting all ops after (but not including) 18
I20251215 21:06:20.857120 24281 pending_rounds.cc:107] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Aborting uncommitted NO_OP operation due to leader change: 6.19
I20251215 21:06:20.857151 24281 raft_consensus.cc:2889] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251215 21:06:20.857520 24521 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251215 21:06:20.861697 24357 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 7 FOLLOWER]: Refusing update from remote peer 22b7ab1cf11d4d78ba5425555355b386: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 7 index: 19. (index mismatch)
I20251215 21:06:20.862026 24521 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251215 21:06:20.962867 24491 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 22b7ab1cf11d4d78ba5425555355b386)
I20251215 21:06:20.963027 24491 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 7 FOLLOWER]: Advancing to term 8
I20251215 21:06:20.963943 24491 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.964352 24491 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 8 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:20.964751 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:20.964758 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 8 candidate_status { last_received { term: 7 index: 19 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:20.964918 24281 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 7 FOLLOWER]: Advancing to term 8
I20251215 21:06:20.965675 24281 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a30b1f25d743439f4b73d5389c1017 in term 8.
I20251215 21:06:20.965937 24323 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017; no voters: 22b7ab1cf11d4d78ba5425555355b386
I20251215 21:06:20.966099 24491 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 FOLLOWER]: Leader election won for term 8
I20251215 21:06:20.966202 24491 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 LEADER]: Becoming Leader. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Running, Role: LEADER
I20251215 21:06:20.966347 24491 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 7.19, Last appended by leader: 19, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:20.967579 24178 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 reported cstate change: term changed from 7 to 8, leader changed from 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67) to a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66). New cstate: current_term: 8 leader_uuid: "a1a30b1f25d743439f4b73d5389c1017" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:20.974129 24357 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 LEADER]: Rejecting Update request from peer 22b7ab1cf11d4d78ba5425555355b386 for earlier term 7. Current term is 8. Ops: []
I20251215 21:06:20.974449 24484 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: INVALID_TERM, Last received: 7.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251215 21:06:20.974560 24484 consensus_queue.cc:1243] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Peer a1a30b1f25d743439f4b73d5389c1017 log is divergent from this leader: its last log entry 8.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20251215 21:06:20.978983 24281 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 8 FOLLOWER]: Rejecting Update request from peer 22b7ab1cf11d4d78ba5425555355b386 for earlier term 7. Current term is 8. Ops: []
I20251215 21:06:20.979364 24484 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: INVALID_TERM, Last received: 7.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251215 21:06:21.062592 24281 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 8 FOLLOWER]: Refusing update from remote peer a1a30b1f25d743439f4b73d5389c1017: Log matching property violated. Preceding OpId in replica: term: 7 index: 19. Preceding OpId from leader: term: 8 index: 20. (index mismatch)
I20251215 21:06:21.062953 24491 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251215 21:06:21.075384 24281 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 8 FOLLOWER]: Rejecting Update request from peer 22b7ab1cf11d4d78ba5425555355b386 for earlier term 7. Current term is 8. Ops: []
I20251215 21:06:21.075734 24521 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: INVALID_TERM, Last received: 7.19, Next index: 20, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.090736 24432 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 LEADER]: Stepping down as leader of term 7
I20251215 21:06:21.090831 24432 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22b7ab1cf11d4d78ba5425555355b386, State: Running, Role: LEADER
I20251215 21:06:21.090976 24432 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 7.19, Last appended by leader: 19, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.091173 24432 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 7 FOLLOWER]: Advancing to term 8
I20251215 21:06:21.092124 24432 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 8 FOLLOWER]: Refusing update from remote peer a1a30b1f25d743439f4b73d5389c1017: Log matching property violated. Preceding OpId in replica: term: 7 index: 19. Preceding OpId from leader: term: 8 index: 20. (index mismatch)
I20251215 21:06:21.092504 24491 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251215 21:06:21.165791 24522 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 8 FOLLOWER]: Starting leader election (detected failure of leader a1a30b1f25d743439f4b73d5389c1017)
I20251215 21:06:21.165903 24522 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 8 FOLLOWER]: Advancing to term 9
I20251215 21:06:21.166785 24522 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.167301 24522 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 9 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:21.167673 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:21.167743 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:21.168049 24245 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc; no voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:21.168196 24522 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251215 21:06:21.186239 24281 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 9 FOLLOWER]: Rejecting Update request from peer a1a30b1f25d743439f4b73d5389c1017 for earlier term 8. Current term is 9. Ops: []
I20251215 21:06:21.186605 24516 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.189831 24516 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 LEADER]: Stepping down as leader of term 8
I20251215 21:06:21.189888 24516 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a1a30b1f25d743439f4b73d5389c1017, State: Running, Role: LEADER
I20251215 21:06:21.190011 24516 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.190160 24516 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 8 FOLLOWER]: Advancing to term 9
I20251215 21:06:21.296888 24522 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:21.297014 24522 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 9 FOLLOWER]: Advancing to term 10
I20251215 21:06:21.297894 24522 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.298264 24522 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 10 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:21.298681 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 10 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:21.298710 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 10 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:21.298782 24432 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 8 FOLLOWER]: Advancing to term 10
I20251215 21:06:21.298844 24357 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 9 FOLLOWER]: Advancing to term 10
I20251215 21:06:21.299504 24432 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290befd6e2fa43ad9a5b326619654cfc in term 10.
I20251215 21:06:21.299602 24357 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290befd6e2fa43ad9a5b326619654cfc in term 10.
I20251215 21:06:21.299784 24245 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc; no voters:
I20251215 21:06:21.299965 24522 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 10 FOLLOWER]: Leader election won for term 10
I20251215 21:06:21.300038 24522 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 10 LEADER]: Becoming Leader. State: Replica: 290befd6e2fa43ad9a5b326619654cfc, State: Running, Role: LEADER
I20251215 21:06:21.300170 24522 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.301443 24177 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc reported cstate change: term changed from 8 to 10, leader changed from a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66) to 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65). New cstate: current_term: 10 leader_uuid: "290befd6e2fa43ad9a5b326619654cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:21.395679 24432 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 10 FOLLOWER]: Refusing update from remote peer 290befd6e2fa43ad9a5b326619654cfc: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 10 index: 21. (index mismatch)
I20251215 21:06:21.396051 24522 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.415665 24516 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:21.415782 24516 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 10 FOLLOWER]: Advancing to term 11
I20251215 21:06:21.416644 24516 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.417052 24516 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 11 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:21.417428 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 11 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:21.417433 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 11 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:21.417761 24323 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1a30b1f25d743439f4b73d5389c1017; no voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc
I20251215 21:06:21.417917 24516 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251215 21:06:21.421339 24357 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 10. Current term is 11. Ops: []
I20251215 21:06:21.421661 24522 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.465695 24357 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 10. Current term is 11. Ops: [10.21-10.21]
I20251215 21:06:21.466114 24522 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.499171 24518 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 290befd6e2fa43ad9a5b326619654cfc)
I20251215 21:06:21.499307 24518 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 10 FOLLOWER]: Advancing to term 11
I20251215 21:06:21.500193 24518 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.500576 24518 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 11 election: Requested vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:21.500986 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:21.500986 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 11 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:21.501163 24357 raft_consensus.cc:2393] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 22b7ab1cf11d4d78ba5425555355b386 in current term 11: Already voted for candidate a1a30b1f25d743439f4b73d5389c1017 in this term.
I20251215 21:06:21.501389 24398 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386; no voters: 290befd6e2fa43ad9a5b326619654cfc, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:21.501538 24518 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251215 21:06:21.519344 24516 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:21.519496 24516 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 11 FOLLOWER]: Advancing to term 12
I20251215 21:06:21.520365 24516 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.520776 24516 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 12 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:21.521196 24432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 12 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:21.521303 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 12 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:21.521313 24432 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 11 FOLLOWER]: Advancing to term 12
I20251215 21:06:21.522301 24432 raft_consensus.cc:2410] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a1a30b1f25d743439f4b73d5389c1017 for term 12 because replica has last-logged OpId of term: 10 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 20.
I20251215 21:06:21.522454 24431 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 12 FOLLOWER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 10. Current term is 12. Ops: []
I20251215 21:06:21.522584 24321 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1a30b1f25d743439f4b73d5389c1017; no voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc
I20251215 21:06:21.522795 24516 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251215 21:06:21.522825 24524 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: INVALID_TERM, Last received: 10.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251215 21:06:21.582363 24357 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 12 FOLLOWER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 10. Current term is 12. Ops: [10.21-10.21]
I20251215 21:06:21.582769 24485 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.609789 24431 raft_consensus.cc:1240] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 12 FOLLOWER]: Rejecting Update request from peer 290befd6e2fa43ad9a5b326619654cfc for earlier term 10. Current term is 12. Ops: []
I20251215 21:06:21.610204 24522 consensus_queue.cc:1059] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 }, Status: INVALID_TERM, Last received: 10.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251215 21:06:21.617393 24524 raft_consensus.cc:3055] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 10 LEADER]: Stepping down as leader of term 10
I20251215 21:06:21.617486 24524 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 290befd6e2fa43ad9a5b326619654cfc, State: Running, Role: LEADER
I20251215 21:06:21.617655 24524 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 10.21, Last appended by leader: 21, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.617872 24524 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 10 FOLLOWER]: Advancing to term 11
I20251215 21:06:21.618935 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.620153 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.622342 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.626516 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.634701 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.647975 24518 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:21.648097 24518 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 12 FOLLOWER]: Advancing to term 13
I20251215 21:06:21.649044 24518 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.649453 24518 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 13 election: Requested vote from peers 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:21.649824 24281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 13 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "290befd6e2fa43ad9a5b326619654cfc"
I20251215 21:06:21.649832 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "22b7ab1cf11d4d78ba5425555355b386" candidate_term: 13 candidate_status { last_received { term: 10 index: 21 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:21.649933 24281 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 11 FOLLOWER]: Advancing to term 13
I20251215 21:06:21.649976 24357 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 12 FOLLOWER]: Advancing to term 13
I20251215 21:06:21.650645 24281 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b7ab1cf11d4d78ba5425555355b386 in term 13.
I20251215 21:06:21.650749 24357 raft_consensus.cc:2468] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b7ab1cf11d4d78ba5425555355b386 in term 13.
I20251215 21:06:21.650906 24398 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc; no voters:
I20251215 21:06:21.650949 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.651060 24518 raft_consensus.cc:2804] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 13 FOLLOWER]: Leader election won for term 13
I20251215 21:06:21.651209 24518 raft_consensus.cc:697] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 13 LEADER]: Becoming Leader. State: Replica: 22b7ab1cf11d4d78ba5425555355b386, State: Running, Role: LEADER
I20251215 21:06:21.651348 24518 consensus_queue.cc:237] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 10.21, Last appended by leader: 21, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:21.652616 24177 catalog_manager.cc:5654] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 reported cstate change: term changed from 10 to 13, leader changed from 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65) to 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67). New cstate: current_term: 13 leader_uuid: "22b7ab1cf11d4d78ba5425555355b386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20251215 21:06:21.683209 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.747570 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.755702 24281 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 13 FOLLOWER]: Refusing update from remote peer 22b7ab1cf11d4d78ba5425555355b386: Log matching property violated. Preceding OpId in replica: term: 10 index: 21. Preceding OpId from leader: term: 13 index: 22. (index mismatch)
I20251215 21:06:21.756103 24518 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251215 21:06:21.758378 24357 raft_consensus.cc:1275] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 13 FOLLOWER]: Refusing update from remote peer 22b7ab1cf11d4d78ba5425555355b386: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 13 index: 22. (index mismatch)
I20251215 21:06:21.758952 24518 consensus_queue.cc:1048] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:21.875929 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:21.876194 23617 log_reader.cc:385] T 3c998c8faa354807b3ef99adc3b97252: removed 1 log segments from log reader
I20251215 21:06:21.876246 23617 log.cc:1079] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Deleting log segment in path: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root/wals/3c998c8faa354807b3ef99adc3b97252/wal-000000001 (ops 1-16)
I20251215 21:06:21.877548 23617 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20251215 21:06:21.877671 23617 master.cc:561] Master@127.23.16.126:44931 shutting down...
I20251215 21:06:22.016013 23617 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:06:22.016214 23617 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:22.016288 23617 tablet_replica.cc:333] T 00000000000000000000000000000000 P a0af3555a77c41f7bc301b5e61c48f5e: stopping tablet replica
I20251215 21:06:22.028894 23617 master.cc:583] Master@127.23.16.126:44931 shutdown complete.
I20251215 21:06:22.031399 23617 tablet_server.cc:178] TabletServer@127.23.16.65:0 shutting down...
I20251215 21:06:22.034574 24524 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 13 FOLLOWER]: Starting leader election (detected failure of leader 22b7ab1cf11d4d78ba5425555355b386)
I20251215 21:06:22.034660 24524 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 13 FOLLOWER]: Advancing to term 14
I20251215 21:06:22.035461 24524 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:22.035792 24524 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 14 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
I20251215 21:06:22.036175 24431 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
I20251215 21:06:22.036175 24357 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "290befd6e2fa43ad9a5b326619654cfc" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "a1a30b1f25d743439f4b73d5389c1017"
I20251215 21:06:22.036571 24245 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc; no voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:22.036708 24524 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20251215 21:06:22.040987 24398 consensus_peers.cc:597] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 -> Peer 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859): Couldn't send request to peer 290befd6e2fa43ad9a5b326619654cfc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251215 21:06:22.085354 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:22.085595 23617 tablet_replica.cc:333] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: stopping tablet replica
I20251215 21:06:22.085712 23617 raft_consensus.cc:2243] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:22.085876 23617 raft_consensus.cc:2272] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:22.098743 23617 tablet_server.cc:195] TabletServer@127.23.16.65:0 shutdown complete.
I20251215 21:06:22.100782 23617 tablet_server.cc:178] TabletServer@127.23.16.66:0 shutting down...
I20251215 21:06:22.105787 24516 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 22b7ab1cf11d4d78ba5425555355b386)
I20251215 21:06:22.105875 24516 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 13 FOLLOWER]: Advancing to term 14
I20251215 21:06:22.106649 24516 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:22.107012 24516 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 14 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859)
I20251215 21:06:22.107359 24431 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c998c8faa354807b3ef99adc3b97252" candidate_uuid: "a1a30b1f25d743439f4b73d5389c1017" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "22b7ab1cf11d4d78ba5425555355b386"
W20251215 21:06:22.108107 24323 leader_election.cc:336] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 290befd6e2fa43ad9a5b326619654cfc (127.23.16.65:44859): Network error: Client connection negotiation failed: client connection to 127.23.16.65:44859: connect: Connection refused (error 111)
I20251215 21:06:22.108172 24323 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1a30b1f25d743439f4b73d5389c1017; no voters: 22b7ab1cf11d4d78ba5425555355b386, 290befd6e2fa43ad9a5b326619654cfc
I20251215 21:06:22.108289 24516 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20251215 21:06:22.108623 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:22.108768 23617 tablet_replica.cc:333] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017: stopping tablet replica
I20251215 21:06:22.108892 23617 raft_consensus.cc:2243] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 14 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:22.108981 23617 raft_consensus.cc:2272] T 3c998c8faa354807b3ef99adc3b97252 P a1a30b1f25d743439f4b73d5389c1017 [term 14 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:22.121379 23617 tablet_server.cc:195] TabletServer@127.23.16.66:0 shutdown complete.
W20251215 21:06:22.122083 24396 consensus_peers.cc:597] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 -> Peer a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861): Couldn't send request to peer a1a30b1f25d743439f4b73d5389c1017. Status: Network error: Client connection negotiation failed: client connection to 127.23.16.66:39861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251215 21:06:22.123229 23617 tablet_server.cc:178] TabletServer@127.23.16.67:0 shutting down...
I20251215 21:06:22.127215 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:22.127378 23617 tablet_replica.cc:333] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386: stopping tablet replica
I20251215 21:06:22.127478 23617 raft_consensus.cc:2243] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 13 LEADER]: Raft consensus shutting down.
I20251215 21:06:22.127635 23617 raft_consensus.cc:2272] T 3c998c8faa354807b3ef99adc3b97252 P 22b7ab1cf11d4d78ba5425555355b386 [term 13 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:22.140017 23617 tablet_server.cc:195] TabletServer@127.23.16.67:0 shutdown complete.
I20251215 21:06:22.141767 23617 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20251215 21:06:22.142000 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:22.143477 24535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.143553 24533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.143608 24532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.143762 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:22.143909 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:22.143944 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:22.143961 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832782143961 us; error 0 us; skew 500 ppm
I20251215 21:06:22.144685 23617 webserver.cc:492] Webserver started at http://127.23.16.65:39799/ using document root <none> and password file <none>
I20251215 21:06:22.144843 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:22.144881 23617 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251215 21:06:22.145879 23617 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.148531 24544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.148734 23617 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:06:22.148815 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "290befd6e2fa43ad9a5b326619654cfc"
format_stamp: "Formatted at 2025-12-15 21:06:19 on dist-test-slave-bslw"
I20251215 21:06:22.148895 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5299
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:22.155764 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:22.156037 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:22.156265 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:22.156890 24551 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251215 21:06:22.158272 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251215 21:06:22.158324 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251215 21:06:22.158372 23617 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251215 21:06:22.158988 23617 ts_tablet_manager.cc:616] Registered 1 tablets
I20251215 21:06:22.159025 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251215 21:06:22.159075 24551 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Bootstrap starting.
I20251215 21:06:22.163156 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:44859
I20251215 21:06:22.163739 24551 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:06:22.164175 24551 tablet_bootstrap.cc:492] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Bootstrap complete.
I20251215 21:06:22.164399 24551 ts_tablet_manager.cc:1403] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:06:22.165014 24551 raft_consensus.cc:359] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:22.165189 24551 raft_consensus.cc:740] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290befd6e2fa43ad9a5b326619654cfc, State: Initialized, Role: FOLLOWER
I20251215 21:06:22.165412 24551 consensus_queue.cc:260] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 13.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:22.165892 24551 ts_tablet_manager.cc:1434] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251215 21:06:22.165971 24609 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:44859 every 8 connection(s)
W20251215 21:06:22.167156 24610 heartbeater.cc:646] Failed to heartbeat to 127.23.16.126:44931 (0 consecutive failures): Network error: Failed to ping master at 127.23.16.126:44931: Client connection negotiation failed: client connection to 127.23.16.126:44931: connect: Connection refused (error 111)
W20251215 21:06:22.168074 24610 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251215 21:06:22.177253 23617 tablet_server.cc:178] TabletServer@127.23.16.65:44859 shutting down...
I20251215 21:06:22.274080 24613 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:22.274230 24613 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 14 FOLLOWER]: Advancing to term 15
I20251215 21:06:22.275192 24613 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:22.275656 24613 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 15 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
W20251215 21:06:22.276866 24546 leader_election.cc:336] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457): Network error: Client connection negotiation failed: client connection to 127.23.16.67:33457: connect: Connection refused (error 111)
W20251215 21:06:22.276985 24546 leader_election.cc:336] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861): Network error: Client connection negotiation failed: client connection to 127.23.16.66:39861: connect: Connection refused (error 111)
I20251215 21:06:22.277030 24546 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc; no voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017
I20251215 21:06:22.277155 24613 raft_consensus.cc:2749] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20251215 21:06:22.416913 24613 raft_consensus.cc:493] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:22.416978 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:22.417062 24613 raft_consensus.cc:3060] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 15 FOLLOWER]: Advancing to term 16
I20251215 21:06:22.417383 23617 tablet_replica.cc:333] stopping tablet replica
I20251215 21:06:22.418040 24613 raft_consensus.cc:515] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b7ab1cf11d4d78ba5425555355b386" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33457 } } peers { permanent_uuid: "290befd6e2fa43ad9a5b326619654cfc" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 44859 } } peers { permanent_uuid: "a1a30b1f25d743439f4b73d5389c1017" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 39861 } }
I20251215 21:06:22.418239 23617 raft_consensus.cc:2243] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 16 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:22.418387 23617 raft_consensus.cc:2272] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [term 16 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:22.418442 24613 leader_election.cc:290] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 16 election: Requested vote from peers 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457), a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861)
W20251215 21:06:22.419540 24546 leader_election.cc:336] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a1a30b1f25d743439f4b73d5389c1017 (127.23.16.66:39861): Service unavailable: Unable to start connection negotiation thread: The pool has been shut down.
W20251215 21:06:22.419662 24546 leader_election.cc:336] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 22b7ab1cf11d4d78ba5425555355b386 (127.23.16.67:33457): Network error: Client connection negotiation failed: client connection to 127.23.16.67:33457: connect: Connection refused (error 111)
I20251215 21:06:22.419714 24546 leader_election.cc:304] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 290befd6e2fa43ad9a5b326619654cfc; no voters: 22b7ab1cf11d4d78ba5425555355b386, a1a30b1f25d743439f4b73d5389c1017
W20251215 21:06:22.419801 24546 raft_consensus.cc:2686] T 3c998c8faa354807b3ef99adc3b97252 P 290befd6e2fa43ad9a5b326619654cfc: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251215 21:06:22.431308 23617 tablet_server.cc:195] TabletServer@127.23.16.65:44859 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2940 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20251215 21:06:22.437294 23617 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.16.126:37623
I20251215 21:06:22.437599 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:22.438987 24618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.439111 24619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.439131 24621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.439253 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:22.439404 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:22.439441 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:22.439455 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832782439455 us; error 0 us; skew 500 ppm
I20251215 21:06:22.440189 23617 webserver.cc:492] Webserver started at http://127.23.16.126:37591/ using document root <none> and password file <none>
I20251215 21:06:22.440335 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:22.440373 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:22.440443 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:22.440778 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/master-0-root/instance:
uuid: "380db959b1b0490ea9798ec3edf06f1d"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.442137 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.442948 24626 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.443167 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.443224 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/master-0-root
uuid: "380db959b1b0490ea9798ec3edf06f1d"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.443274 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:22.453650 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:22.453944 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:22.457576 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.126:37623
I20251215 21:06:22.457983 24678 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.126:37623 every 8 connection(s)
I20251215 21:06:22.458479 24679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:22.460175 24679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d: Bootstrap starting.
I20251215 21:06:22.460929 24679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:22.461798 24679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d: No bootstrap required, opened a new log
I20251215 21:06:22.462144 24679 raft_consensus.cc:359] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380db959b1b0490ea9798ec3edf06f1d" member_type: VOTER }
I20251215 21:06:22.462224 24679 raft_consensus.cc:385] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:22.462248 24679 raft_consensus.cc:740] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380db959b1b0490ea9798ec3edf06f1d, State: Initialized, Role: FOLLOWER
I20251215 21:06:22.462352 24679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380db959b1b0490ea9798ec3edf06f1d" member_type: VOTER }
I20251215 21:06:22.462420 24679 raft_consensus.cc:399] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:06:22.462448 24679 raft_consensus.cc:493] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:06:22.462481 24679 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:22.463112 24679 raft_consensus.cc:515] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380db959b1b0490ea9798ec3edf06f1d" member_type: VOTER }
I20251215 21:06:22.463214 24679 leader_election.cc:304] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380db959b1b0490ea9798ec3edf06f1d; no voters:
I20251215 21:06:22.463346 24679 leader_election.cc:290] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:06:22.463460 24682 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:06:22.463732 24682 raft_consensus.cc:697] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 1 LEADER]: Becoming Leader. State: Replica: 380db959b1b0490ea9798ec3edf06f1d, State: Running, Role: LEADER
I20251215 21:06:22.463737 24679 sys_catalog.cc:565] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:06:22.463893 24682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380db959b1b0490ea9798ec3edf06f1d" member_type: VOTER }
I20251215 21:06:22.464263 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380db959b1b0490ea9798ec3edf06f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380db959b1b0490ea9798ec3edf06f1d" member_type: VOTER } }
I20251215 21:06:22.464365 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:22.464278 24684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380db959b1b0490ea9798ec3edf06f1d. Latest consensus state: current_term: 1 leader_uuid: "380db959b1b0490ea9798ec3edf06f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380db959b1b0490ea9798ec3edf06f1d" member_type: VOTER } }
I20251215 21:06:22.464615 24688 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:06:22.464591 24684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:22.465359 24688 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:06:22.465528 23617 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 21:06:22.466888 24688 catalog_manager.cc:1357] Generated new cluster ID: abc5ad18cfbf40c496561a2761736066
I20251215 21:06:22.466940 24688 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:06:22.472389 24688 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:06:22.472899 24688 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:06:22.479465 24688 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d: Generated new TSK 0
I20251215 21:06:22.479604 24688 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:06:22.481498 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:22.483155 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.483290 24700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.483285 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:22.483284 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.483592 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:22.483640 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:22.483676 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832782483674 us; error 0 us; skew 500 ppm
I20251215 21:06:22.484462 23617 webserver.cc:492] Webserver started at http://127.23.16.65:33297/ using document root <none> and password file <none>
I20251215 21:06:22.484612 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:22.484663 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:22.484740 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:22.485176 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root/instance:
uuid: "cddb404a5d3c4202a43cbad25a5e55a0"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.486584 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251215 21:06:22.487416 24708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.487648 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.487708 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "cddb404a5d3c4202a43cbad25a5e55a0"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.487763 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:22.517305 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:22.517694 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:22.517940 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:22.518397 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:22.518435 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.518466 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:22.518492 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.522207 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:36265
I20251215 21:06:22.522238 24771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:36265 every 8 connection(s)
I20251215 21:06:22.522912 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:22.524477 24776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.524562 24777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.524672 24779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.524787 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:22.525063 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:22.525117 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:22.525144 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832782525144 us; error 0 us; skew 500 ppm
I20251215 21:06:22.525913 23617 webserver.cc:492] Webserver started at http://127.23.16.66:38741/ using document root <none> and password file <none>
I20251215 21:06:22.526052 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:22.526100 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:22.526167 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:22.526563 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-1-root/instance:
uuid: "b7a9ad9904a94316a4f9c6dda5598496"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.527238 24772 heartbeater.cc:344] Connected to a master server at 127.23.16.126:37623
I20251215 21:06:22.527324 24772 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:22.527526 24772 heartbeater.cc:507] Master 127.23.16.126:37623 requested a full tablet report, sending...
I20251215 21:06:22.528096 24643 ts_manager.cc:194] Registered new tserver with Master: cddb404a5d3c4202a43cbad25a5e55a0 (127.23.16.65:36265)
I20251215 21:06:22.528826 24643 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59822
I20251215 21:06:22.529274 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:06:22.530164 24784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.530382 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.530440 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-1-root
uuid: "b7a9ad9904a94316a4f9c6dda5598496"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.530507 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:22.549433 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:22.549813 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:22.550081 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:22.550496 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:22.550532 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.550565 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:22.550593 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.555114 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.66:32939
I20251215 21:06:22.555177 24847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.66:32939 every 8 connection(s)
I20251215 21:06:22.555840 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:22.557579 24851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:22.557787 24854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.557828 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:22.557850 24852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:22.558167 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:22.558215 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:22.558244 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832782558243 us; error 0 us; skew 500 ppm
I20251215 21:06:22.559206 23617 webserver.cc:492] Webserver started at http://127.23.16.67:42531/ using document root <none> and password file <none>
I20251215 21:06:22.559365 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:22.559437 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:22.559517 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:22.559926 24848 heartbeater.cc:344] Connected to a master server at 127.23.16.126:37623
I20251215 21:06:22.559957 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-2-root/instance:
uuid: "25b5a2d0704e42e8b47089b0f33e3769"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.560005 24848 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:22.560217 24848 heartbeater.cc:507] Master 127.23.16.126:37623 requested a full tablet report, sending...
I20251215 21:06:22.560776 24643 ts_manager.cc:194] Registered new tserver with Master: b7a9ad9904a94316a4f9c6dda5598496 (127.23.16.66:32939)
I20251215 21:06:22.561342 24643 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59832
I20251215 21:06:22.561543 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251215 21:06:22.562450 24859 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.562647 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:06:22.562706 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-2-root
uuid: "25b5a2d0704e42e8b47089b0f33e3769"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:22.562770 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:22.573848 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:22.574213 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:22.574481 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:22.574874 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:22.574911 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.574951 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:22.574978 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:22.579104 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.67:33629
I20251215 21:06:22.579216 24922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.67:33629 every 8 connection(s)
I20251215 21:06:22.583076 24923 heartbeater.cc:344] Connected to a master server at 127.23.16.126:37623
I20251215 21:06:22.583170 24923 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:22.583360 24923 heartbeater.cc:507] Master 127.23.16.126:37623 requested a full tablet report, sending...
I20251215 21:06:22.583922 24643 ts_manager.cc:194] Registered new tserver with Master: 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67:33629)
I20251215 21:06:22.583993 23617 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004521738s
I20251215 21:06:22.584479 24643 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59838
I20251215 21:06:22.593487 23617 test_util.cc:276] Using random seed: -71483956
I20251215 21:06:22.599804 24643 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59880:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:06:22.600255 24643 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:06:22.609102 24812 tablet_service.cc:1505] Processing CreateTablet for tablet 6f10c102df4e48f9aa4d81b115e7df4e (DEFAULT_TABLE table=test-workload [id=ad111e7adcbb435bada1e331de77bef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:22.609427 24812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f10c102df4e48f9aa4d81b115e7df4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:22.609440 24887 tablet_service.cc:1505] Processing CreateTablet for tablet 6f10c102df4e48f9aa4d81b115e7df4e (DEFAULT_TABLE table=test-workload [id=ad111e7adcbb435bada1e331de77bef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:22.609529 24736 tablet_service.cc:1505] Processing CreateTablet for tablet 6f10c102df4e48f9aa4d81b115e7df4e (DEFAULT_TABLE table=test-workload [id=ad111e7adcbb435bada1e331de77bef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:22.609711 24887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f10c102df4e48f9aa4d81b115e7df4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:22.609711 24736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f10c102df4e48f9aa4d81b115e7df4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:22.611904 24945 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Bootstrap starting.
I20251215 21:06:22.611984 24944 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496: Bootstrap starting.
I20251215 21:06:22.612855 24945 tablet_bootstrap.cc:654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:22.612854 24944 tablet_bootstrap.cc:654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:22.613559 24946 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769: Bootstrap starting.
I20251215 21:06:22.613898 24944 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496: No bootstrap required, opened a new log
I20251215 21:06:22.613921 24945 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: No bootstrap required, opened a new log
I20251215 21:06:22.614024 24944 ts_tablet_manager.cc:1403] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251215 21:06:22.614105 24945 ts_tablet_manager.cc:1403] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:06:22.614434 24946 tablet_bootstrap.cc:654] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:22.614444 24944 raft_consensus.cc:359] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:22.614569 24944 raft_consensus.cc:385] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:22.614552 24945 raft_consensus.cc:359] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:22.614624 24944 raft_consensus.cc:740] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a9ad9904a94316a4f9c6dda5598496, State: Initialized, Role: FOLLOWER
I20251215 21:06:22.614670 24945 raft_consensus.cc:385] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:22.614737 24945 raft_consensus.cc:740] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddb404a5d3c4202a43cbad25a5e55a0, State: Initialized, Role: FOLLOWER
I20251215 21:06:22.614768 24944 consensus_queue.cc:260] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:22.614892 24945 consensus_queue.cc:260] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:22.615159 24944 ts_tablet_manager.cc:1434] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251215 21:06:22.615214 24945 ts_tablet_manager.cc:1434] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.615356 24848 heartbeater.cc:499] Master 127.23.16.126:37623 was elected leader, sending a full tablet report...
I20251215 21:06:22.615456 24946 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769: No bootstrap required, opened a new log
I20251215 21:06:22.615532 24946 ts_tablet_manager.cc:1403] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:06:22.615813 24772 heartbeater.cc:499] Master 127.23.16.126:37623 was elected leader, sending a full tablet report...
I20251215 21:06:22.615904 24946 raft_consensus.cc:359] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:22.616000 24946 raft_consensus.cc:385] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:22.616207 24946 raft_consensus.cc:740] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b5a2d0704e42e8b47089b0f33e3769, State: Initialized, Role: FOLLOWER
I20251215 21:06:22.616398 24946 consensus_queue.cc:260] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:22.616748 24946 ts_tablet_manager.cc:1434] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:22.616900 24923 heartbeater.cc:499] Master 127.23.16.126:37623 was elected leader, sending a full tablet report...
I20251215 21:06:23.001801 24951 raft_consensus.cc:493] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:06:23.001955 24951 raft_consensus.cc:515] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.002439 24951 leader_election.cc:290] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7a9ad9904a94316a4f9c6dda5598496 (127.23.16.66:32939), 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67:33629)
I20251215 21:06:23.006428 24822 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f10c102df4e48f9aa4d81b115e7df4e" candidate_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9ad9904a94316a4f9c6dda5598496" is_pre_election: true
I20251215 21:06:23.006433 24897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f10c102df4e48f9aa4d81b115e7df4e" candidate_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b5a2d0704e42e8b47089b0f33e3769" is_pre_election: true
I20251215 21:06:23.006565 24822 raft_consensus.cc:2468] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddb404a5d3c4202a43cbad25a5e55a0 in term 0.
I20251215 21:06:23.006597 24897 raft_consensus.cc:2468] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddb404a5d3c4202a43cbad25a5e55a0 in term 0.
I20251215 21:06:23.006875 24709 leader_election.cc:304] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a9ad9904a94316a4f9c6dda5598496, cddb404a5d3c4202a43cbad25a5e55a0; no voters:
I20251215 21:06:23.007031 24951 raft_consensus.cc:2804] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:06:23.007083 24951 raft_consensus.cc:493] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:06:23.007138 24951 raft_consensus.cc:3060] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:23.007934 24951 raft_consensus.cc:515] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.008242 24951 leader_election.cc:290] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [CANDIDATE]: Term 1 election: Requested vote from peers b7a9ad9904a94316a4f9c6dda5598496 (127.23.16.66:32939), 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67:33629)
I20251215 21:06:23.008554 24822 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f10c102df4e48f9aa4d81b115e7df4e" candidate_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a9ad9904a94316a4f9c6dda5598496"
I20251215 21:06:23.008612 24897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f10c102df4e48f9aa4d81b115e7df4e" candidate_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b5a2d0704e42e8b47089b0f33e3769"
I20251215 21:06:23.008661 24822 raft_consensus.cc:3060] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:23.008709 24897 raft_consensus.cc:3060] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:23.009490 24897 raft_consensus.cc:2468] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddb404a5d3c4202a43cbad25a5e55a0 in term 1.
I20251215 21:06:23.009590 24822 raft_consensus.cc:2468] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddb404a5d3c4202a43cbad25a5e55a0 in term 1.
I20251215 21:06:23.009732 24709 leader_election.cc:304] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b5a2d0704e42e8b47089b0f33e3769, cddb404a5d3c4202a43cbad25a5e55a0; no voters:
I20251215 21:06:23.009891 24951 raft_consensus.cc:2804] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:06:23.010054 24951 raft_consensus.cc:697] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Becoming Leader. State: Replica: cddb404a5d3c4202a43cbad25a5e55a0, State: Running, Role: LEADER
I20251215 21:06:23.010205 24951 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.011651 24642 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cddb404a5d3c4202a43cbad25a5e55a0 (127.23.16.65). New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.060413 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:06:23.060413 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:06:23.060933 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:06:23.061100 24951 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:06:23.104813 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.105774 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20251215 21:06:23.105774 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20251215 21:06:23.106155 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251215 21:06:23.106319 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251215 21:06:23.107633 24978 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index -1 to 18, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.107913 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index -1 to 18, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.107964 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.18: config changed from index -1 to 18, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.110234 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 reported cstate change: config changed from index -1 to 18, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.111214 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.112154 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20251215 21:06:23.112154 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20251215 21:06:23.112648 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251215 21:06:23.112927 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20251215 21:06:23.113507 24640 catalog_manager.cc:5332] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 18: no extra replica candidate found for tablet 6f10c102df4e48f9aa4d81b115e7df4e (table test-workload [id=ad111e7adcbb435bada1e331de77bef9]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20251215 21:06:23.114249 24979 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.114744 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.115832 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.116389 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.116360 24640 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 18 to 19, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.118316 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20251215 21:06:23.118345 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20251215 21:06:23.119508 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251215 21:06:23.119732 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251215 21:06:23.120745 24954 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.121223 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.121233 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.123015 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 19 to 20, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.123413 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.124323 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20251215 21:06:23.124526 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20251215 21:06:23.125281 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:23.125495 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251215 21:06:23.126658 24951 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.127067 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.127067 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.128391 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.128472 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 20 to 21, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.129633 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20251215 21:06:23.130122 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20251215 21:06:23.130689 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251215 21:06:23.130848 24951 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251215 21:06:23.132140 24951 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.132519 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.133764 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.134047 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.134096 24643 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 21 to 22, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.135257 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20251215 21:06:23.135317 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20251215 21:06:23.135600 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20251215 21:06:23.136458 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20251215 21:06:23.136865 24951 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.138188 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.138736 24643 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 22 to 23, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.138885 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.140414 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.141566 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20251215 21:06:23.141901 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20251215 21:06:23.142285 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20251215 21:06:23.142590 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20251215 21:06:23.143105 24978 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.143474 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.144249 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.145457 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 reported cstate change: config changed from index 23 to 24, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.145848 24628 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 20: aborting the task: latest config opid_index 23; task opid_index 20
I20251215 21:06:23.146374 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.147939 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20251215 21:06:23.147956 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20251215 21:06:23.148241 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20251215 21:06:23.148389 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20251215 21:06:23.148922 24630 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 18: aborting the task: latest config opid_index 24; task opid_index 18
I20251215 21:06:23.149629 24978 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.150012 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.150158 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.151449 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.151902 24640 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 24 to 25, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.152343 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251215 21:06:23.153002 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251215 21:06:23.153211 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251215 21:06:23.153502 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251215 21:06:23.154763 24954 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.155129 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.156296 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.156635 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.156904 24643 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 reported cstate change: config changed from index 25 to 26, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.157941 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251215 21:06:23.158241 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251215 21:06:23.158215 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251215 21:06:23.159327 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251215 21:06:23.159667 24951 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.160053 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.161358 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.161805 24640 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 reported cstate change: config changed from index 26 to 27, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.162312 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251215 21:06:23.162358 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251215 21:06:23.162665 24951 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251215 21:06:23.162763 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20251215 21:06:23.163364 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.164162 24954 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.164583 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.165530 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.166359 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.166486 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 reported cstate change: config changed from index 27 to 28, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.167687 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251215 21:06:23.167728 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251215 21:06:23.168283 24951 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251215 21:06:23.168507 24951 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251215 21:06:23.170065 24978 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.170404 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.170564 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.172266 24640 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 reported cstate change: config changed from index 28 to 29, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.172462 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.173449 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251215 21:06:23.173457 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251215 21:06:23.173799 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251215 21:06:23.173974 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251215 21:06:23.175484 24954 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.175886 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.177116 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.177135 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.177330 24640 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 29 to 30, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.178501 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251215 21:06:23.179253 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251215 21:06:23.179345 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251215 21:06:23.179719 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251215 21:06:23.180727 24979 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.181072 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.182715 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.182727 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 reported cstate change: config changed from index 30 to 31, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.183924 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251215 21:06:23.183971 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251215 21:06:23.184340 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20251215 21:06:23.184604 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251215 21:06:23.184928 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.186333 24979 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.186866 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.187522 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.188102 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.188962 24643 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 reported cstate change: config changed from index 31 to 32, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.189280 24627 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 22: aborting the task: latest config opid_index 31; task opid_index 22
I20251215 21:06:23.189644 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251215 21:06:23.189735 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251215 21:06:23.190042 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251215 21:06:23.190268 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251215 21:06:23.191664 24951 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.192049 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.193545 24640 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 32 to 33, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.193706 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.194784 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20251215 21:06:23.195112 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20251215 21:06:23.195291 24978 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20251215 21:06:23.195514 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251215 21:06:23.195575 24821 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.197646 24978 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.198026 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.199537 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.199656 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.199903 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 33 to 34, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.200827 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20251215 21:06:23.201216 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20251215 21:06:23.201220 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251215 21:06:23.202267 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251215 21:06:23.203122 24954 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.203454 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.203771 24630 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 24: aborting the task: latest config opid_index 34; task opid_index 24
I20251215 21:06:23.205008 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 34 to 35, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.205008 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.205878 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.206415 24822 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20251215 21:06:23.206830 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251215 21:06:23.207163 24897 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20251215 21:06:23.208499 24979 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251215 21:06:23.208675 24954 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.209973 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.210248 24746 consensus_queue.cc:237] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.210330 24643 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 reported cstate change: config changed from index 35 to 36, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:23.210379 24897 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from VOTER to NON_VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: NON_VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.211799 24896 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 LEARNER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20251215 21:06:23.212185 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251215 21:06:23.212985 24821 raft_consensus.cc:1275] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Refusing update from remote peer cddb404a5d3c4202a43cbad25a5e55a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20251215 21:06:23.214080 24954 consensus_queue.cc:1048] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251215 21:06:23.214277 24979 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.214905 24822 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.215420 24896 raft_consensus.cc:2955] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.215647 24628 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 30: aborting the task: latest config opid_index 36; task opid_index 30
I20251215 21:06:23.216732 24641 catalog_manager.cc:5654] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 reported cstate change: config changed from index 36 to 37, 25b5a2d0704e42e8b47089b0f33e3769 (127.23.16.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } } }
I20251215 21:06:23.217021 23617 timestamp_advancement-itest.cc:267] GCing logs...
I20251215 21:06:23.217145 23617 log_reader.cc:385] T 6f10c102df4e48f9aa4d81b115e7df4e: removed 1 log segments from log reader
I20251215 21:06:23.217186 23617 log.cc:1079] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Deleting log segment in path: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root/wals/6f10c102df4e48f9aa4d81b115e7df4e/wal-000000001 (ops 1-17)
I20251215 21:06:23.218523 24629 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 26: aborting the task: latest config opid_index 37; task opid_index 26
I20251215 21:06:23.220041 23617 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20251215 21:06:23.220114 23617 master.cc:561] Master@127.23.16.126:37623 shutting down...
I20251215 21:06:23.235569 24627 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 34: aborting the task: latest config opid_index 37; task opid_index 34
I20251215 21:06:23.257004 24629 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 32: aborting the task: latest config opid_index 37; task opid_index 32
I20251215 21:06:23.267334 24629 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f10c102df4e48f9aa4d81b115e7df4e with cas_config_opid_index 36: aborting the task: latest config opid_index 37; task opid_index 36
I20251215 21:06:23.455365 23617 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:06:23.455569 23617 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:23.455637 23617 tablet_replica.cc:333] T 00000000000000000000000000000000 P 380db959b1b0490ea9798ec3edf06f1d: stopping tablet replica
I20251215 21:06:23.468372 23617 master.cc:583] Master@127.23.16.126:37623 shutdown complete.
I20251215 21:06:23.470849 23617 tablet_server.cc:178] TabletServer@127.23.16.65:0 shutting down...
I20251215 21:06:23.523474 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:23.523748 23617 tablet_replica.cc:333] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: stopping tablet replica
I20251215 21:06:23.523874 23617 raft_consensus.cc:2243] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:06:23.524108 23617 raft_consensus.cc:2272] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:23.537712 23617 tablet_server.cc:195] TabletServer@127.23.16.65:0 shutdown complete.
I20251215 21:06:23.539865 23617 tablet_server.cc:178] TabletServer@127.23.16.66:0 shutting down...
I20251215 21:06:23.556391 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:23.556596 23617 tablet_replica.cc:333] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496: stopping tablet replica
I20251215 21:06:23.556702 23617 raft_consensus.cc:2243] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:23.556931 23617 raft_consensus.cc:2272] T 6f10c102df4e48f9aa4d81b115e7df4e P b7a9ad9904a94316a4f9c6dda5598496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:23.569571 23617 tablet_server.cc:195] TabletServer@127.23.16.66:0 shutdown complete.
I20251215 21:06:23.571343 23617 tablet_server.cc:178] TabletServer@127.23.16.67:0 shutting down...
I20251215 21:06:23.580271 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:23.580459 23617 tablet_replica.cc:333] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769: stopping tablet replica
I20251215 21:06:23.580562 23617 raft_consensus.cc:2243] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:23.580694 23617 raft_consensus.cc:2272] T 6f10c102df4e48f9aa4d81b115e7df4e P 25b5a2d0704e42e8b47089b0f33e3769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:23.593322 23617 tablet_server.cc:195] TabletServer@127.23.16.67:0 shutdown complete.
I20251215 21:06:23.595060 23617 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20251215 21:06:23.595300 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:23.596789 24982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:23.596938 24981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:23.596964 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:23.597035 24984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:23.597180 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:23.597223 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:23.597237 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832783597237 us; error 0 us; skew 500 ppm
I20251215 21:06:23.597963 23617 webserver.cc:492] Webserver started at http://127.23.16.65:33297/ using document root <none> and password file <none>
I20251215 21:06:23.598099 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:23.598137 23617 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251215 21:06:23.599128 23617 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:23.601590 24991 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:23.601876 23617 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:06:23.601936 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "cddb404a5d3c4202a43cbad25a5e55a0"
format_stamp: "Formatted at 2025-12-15 21:06:22 on dist-test-slave-bslw"
I20251215 21:06:23.601998 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5320
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:23.608920 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:23.609225 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:23.609447 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:23.609972 24998 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251215 21:06:23.611014 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251215 21:06:23.611057 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251215 21:06:23.611092 23617 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251215 21:06:23.611716 23617 ts_tablet_manager.cc:616] Registered 1 tablets
I20251215 21:06:23.611752 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251215 21:06:23.611809 24998 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Bootstrap starting.
I20251215 21:06:23.616273 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:36265
I20251215 21:06:23.617668 25056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:36265 every 8 connection(s)
I20251215 21:06:23.619983 24998 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:06:23.620277 24998 tablet_bootstrap.cc:492] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Bootstrap complete.
W20251215 21:06:23.620540 25057 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251215 21:06:23.620605 24998 ts_tablet_manager.cc:1403] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20251215 21:06:23.621057 24998 raft_consensus.cc:359] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.621173 24998 raft_consensus.cc:740] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddb404a5d3c4202a43cbad25a5e55a0, State: Initialized, Role: FOLLOWER
I20251215 21:06:23.621304 24998 consensus_queue.cc:260] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "b7a9ad9904a94316a4f9c6dda5598496" member_type: VOTER last_known_addr { host: "127.23.16.66" port: 32939 } } peers { permanent_uuid: "25b5a2d0704e42e8b47089b0f33e3769" member_type: VOTER last_known_addr { host: "127.23.16.67" port: 33629 } } peers { permanent_uuid: "cddb404a5d3c4202a43cbad25a5e55a0" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 36265 } }
I20251215 21:06:23.621629 24998 ts_tablet_manager.cc:1434] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20251215 21:06:23.625968 25010 tablet_service.cc:3012] Rejecting scan request for tablet 6f10c102df4e48f9aa4d81b115e7df4e: Uninitialized: clean time has not yet been initialized
I20251215 21:06:23.626379 23617 timestamp_advancement-itest.cc:447] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20251215 21:06:23.626515 23617 tablet_server.cc:178] TabletServer@127.23.16.65:36265 shutting down...
I20251215 21:06:23.868279 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:23.868568 23617 tablet_replica.cc:333] stopping tablet replica
I20251215 21:06:23.868674 23617 raft_consensus.cc:2243] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:06:23.868785 23617 raft_consensus.cc:2272] T 6f10c102df4e48f9aa4d81b115e7df4e P cddb404a5d3c4202a43cbad25a5e55a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:23.881301 23617 tablet_server.cc:195] TabletServer@127.23.16.65:36265 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (1450 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20251215 21:06:23.887440 23617 timestamp_advancement-itest.cc:460] Running with 1 replicas
I20251215 21:06:23.887672 23617 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.16.126:46627
I20251215 21:06:23.888068 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:23.889704 25067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:23.889768 25065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:23.889870 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:23.889870 25064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:23.890244 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:23.890293 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:23.890306 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832783890306 us; error 0 us; skew 500 ppm
I20251215 21:06:23.891029 23617 webserver.cc:492] Webserver started at http://127.23.16.126:35341/ using document root <none> and password file <none>
I20251215 21:06:23.891175 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:23.891215 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:23.891268 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:23.891602 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root/instance:
uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31"
format_stamp: "Formatted at 2025-12-15 21:06:23 on dist-test-slave-bslw"
I20251215 21:06:23.892947 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251215 21:06:23.893785 25072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:23.894004 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:06:23.894062 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root
uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31"
format_stamp: "Formatted at 2025-12-15 21:06:23 on dist-test-slave-bslw"
I20251215 21:06:23.894120 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:23.908679 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:23.909096 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:23.912962 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.126:46627
I20251215 21:06:23.913435 25124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.126:46627 every 8 connection(s)
I20251215 21:06:23.913945 25125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:23.915658 25125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Bootstrap starting.
I20251215 21:06:23.916410 25125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:23.917439 25125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: No bootstrap required, opened a new log
I20251215 21:06:23.917819 25125 raft_consensus.cc:359] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:23.917913 25125 raft_consensus.cc:385] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:23.917944 25125 raft_consensus.cc:740] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2ff0b3fab04c25bfe31f0957cdbc31, State: Initialized, Role: FOLLOWER
I20251215 21:06:23.918063 25125 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:23.918139 25125 raft_consensus.cc:399] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:06:23.918179 25125 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:06:23.918232 25125 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:23.918907 25125 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:23.919026 25125 leader_election.cc:304] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba2ff0b3fab04c25bfe31f0957cdbc31; no voters:
I20251215 21:06:23.919203 25125 leader_election.cc:290] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:06:23.919309 25128 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:06:23.919507 25128 raft_consensus.cc:697] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 LEADER]: Becoming Leader. State: Replica: ba2ff0b3fab04c25bfe31f0957cdbc31, State: Running, Role: LEADER
I20251215 21:06:23.919627 25125 sys_catalog.cc:565] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:06:23.919641 25128 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:23.920056 25129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER } }
I20251215 21:06:23.920099 25130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba2ff0b3fab04c25bfe31f0957cdbc31. Latest consensus state: current_term: 1 leader_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER } }
I20251215 21:06:23.920214 25129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:23.920235 25130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:23.920687 25134 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:06:23.921530 25134 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:06:23.921788 23617 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 21:06:23.923225 25134 catalog_manager.cc:1357] Generated new cluster ID: 92fa87506efc4403af87a9966379070b
I20251215 21:06:23.923273 25134 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:06:23.939983 25134 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:06:23.940524 25134 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:06:23.945178 25134 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Generated new TSK 0
I20251215 21:06:23.945305 25134 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:06:23.953903 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:23.955533 25146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:23.955652 25147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:23.955672 25149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:23.955775 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:23.955952 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:23.955991 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:23.956012 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832783956011 us; error 0 us; skew 500 ppm
I20251215 21:06:23.956728 23617 webserver.cc:492] Webserver started at http://127.23.16.65:37551/ using document root <none> and password file <none>
I20251215 21:06:23.956907 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:23.956956 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:23.957026 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:23.957365 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root/instance:
uuid: "c57ae1126aa643a48dbd45219c6136f2"
format_stamp: "Formatted at 2025-12-15 21:06:23 on dist-test-slave-bslw"
I20251215 21:06:23.958675 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:23.959465 25154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:23.959664 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:23.959724 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "c57ae1126aa643a48dbd45219c6136f2"
format_stamp: "Formatted at 2025-12-15 21:06:23 on dist-test-slave-bslw"
I20251215 21:06:23.959787 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:23.983914 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:23.984280 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:23.984547 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:23.985035 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:23.985072 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:23.985114 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:23.985142 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:23.988785 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:34323
I20251215 21:06:23.988870 25217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:34323 every 8 connection(s)
I20251215 21:06:23.993103 25218 heartbeater.cc:344] Connected to a master server at 127.23.16.126:46627
I20251215 21:06:23.993191 25218 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:23.993372 25218 heartbeater.cc:507] Master 127.23.16.126:46627 requested a full tablet report, sending...
I20251215 21:06:23.993921 25089 ts_manager.cc:194] Registered new tserver with Master: c57ae1126aa643a48dbd45219c6136f2 (127.23.16.65:34323)
I20251215 21:06:23.994607 25089 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38826
I20251215 21:06:23.994725 23617 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005540515s
I20251215 21:06:24.003865 23617 test_util.cc:276] Using random seed: -70073577
I20251215 21:06:24.010315 25089 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38860:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251215 21:06:24.018006 25182 tablet_service.cc:1505] Processing CreateTablet for tablet 9f979ea1cc6744ee840ec9d2f1cbccf1 (DEFAULT_TABLE table=test-workload [id=db64f46791484d2f9164ead9fe284fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:06:24.018273 25182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f979ea1cc6744ee840ec9d2f1cbccf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:06:24.020118 25236 tablet_bootstrap.cc:492] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Bootstrap starting.
I20251215 21:06:24.020995 25236 tablet_bootstrap.cc:654] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Neither blocks nor log segments found. Creating new log.
I20251215 21:06:24.021883 25236 tablet_bootstrap.cc:492] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: No bootstrap required, opened a new log
I20251215 21:06:24.021955 25236 ts_tablet_manager.cc:1403] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:06:24.022286 25236 raft_consensus.cc:359] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.022368 25236 raft_consensus.cc:385] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:06:24.022400 25236 raft_consensus.cc:740] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57ae1126aa643a48dbd45219c6136f2, State: Initialized, Role: FOLLOWER
I20251215 21:06:24.022514 25236 consensus_queue.cc:260] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.022578 25236 raft_consensus.cc:399] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:06:24.022611 25236 raft_consensus.cc:493] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:06:24.022655 25236 raft_consensus.cc:3060] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:06:24.023351 25236 raft_consensus.cc:515] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.023461 25236 leader_election.cc:304] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57ae1126aa643a48dbd45219c6136f2; no voters:
I20251215 21:06:24.023630 25236 leader_election.cc:290] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:06:24.023721 25238 raft_consensus.cc:2804] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:06:24.023902 25238 raft_consensus.cc:697] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 LEADER]: Becoming Leader. State: Replica: c57ae1126aa643a48dbd45219c6136f2, State: Running, Role: LEADER
I20251215 21:06:24.023964 25236 ts_tablet_manager.cc:1434] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:06:24.023980 25218 heartbeater.cc:499] Master 127.23.16.126:46627 was elected leader, sending a full tablet report...
I20251215 21:06:24.024070 25238 consensus_queue.cc:237] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.025262 25089 catalog_manager.cc:5654] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c57ae1126aa643a48dbd45219c6136f2 (127.23.16.65). New cstate: current_term: 1 leader_uuid: "c57ae1126aa643a48dbd45219c6136f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:24.028013 23617 tablet_server.cc:178] TabletServer@127.23.16.65:0 shutting down...
I20251215 21:06:24.030929 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:24.031149 23617 tablet_replica.cc:333] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: stopping tablet replica
I20251215 21:06:24.031241 23617 raft_consensus.cc:2243] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:06:24.031325 23617 raft_consensus.cc:2272] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:24.043990 23617 tablet_server.cc:195] TabletServer@127.23.16.65:0 shutdown complete.
I20251215 21:06:24.046011 23617 master.cc:561] Master@127.23.16.126:46627 shutting down...
I20251215 21:06:24.049177 23617 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:06:24.049331 23617 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:24.049438 23617 tablet_replica.cc:333] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: stopping tablet replica
I20251215 21:06:24.063004 23617 master.cc:583] Master@127.23.16.126:46627 shutdown complete.
I20251215 21:06:24.066350 23617 timestamp_advancement-itest.cc:460] Running with 3 replicas
I20251215 21:06:24.066605 23617 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.16.126:37473
I20251215 21:06:24.066902 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:24.068302 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.068311 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.068440 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:24.068483 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:24.068636 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:24.068672 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:24.068691 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832784068691 us; error 0 us; skew 500 ppm
I20251215 21:06:24.069500 23617 webserver.cc:492] Webserver started at http://127.23.16.126:33241/ using document root <none> and password file <none>
I20251215 21:06:24.069633 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:24.069666 23617 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251215 21:06:24.070580 23617 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:06:24.071305 25251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.071587 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251215 21:06:24.071656 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root
uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31"
format_stamp: "Formatted at 2025-12-15 21:06:23 on dist-test-slave-bslw"
I20251215 21:06:24.071730 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:24.077629 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:24.077935 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:24.081583 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.126:37473
I20251215 21:06:24.082456 25303 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.126:37473 every 8 connection(s)
I20251215 21:06:24.083530 25304 sys_catalog.cc:263] Verifying existing consensus state
I20251215 21:06:24.084173 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Bootstrap starting.
I20251215 21:06:24.089557 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:06:24.089813 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Bootstrap complete.
I20251215 21:06:24.090281 25304 raft_consensus.cc:359] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:24.090373 25304 raft_consensus.cc:740] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2ff0b3fab04c25bfe31f0957cdbc31, State: Initialized, Role: FOLLOWER
I20251215 21:06:24.090464 25304 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:24.090517 25304 raft_consensus.cc:399] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:06:24.090543 25304 raft_consensus.cc:493] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:06:24.090574 25304 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:06:24.091266 25304 raft_consensus.cc:515] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:24.091377 25304 leader_election.cc:304] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba2ff0b3fab04c25bfe31f0957cdbc31; no voters:
I20251215 21:06:24.091538 25304 leader_election.cc:290] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251215 21:06:24.091696 25307 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 21:06:24.091881 25307 raft_consensus.cc:697] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 2 LEADER]: Becoming Leader. State: Replica: ba2ff0b3fab04c25bfe31f0957cdbc31, State: Running, Role: LEADER
I20251215 21:06:24.091905 25304 sys_catalog.cc:565] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:06:24.092022 25307 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER }
I20251215 21:06:24.092499 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER } }
I20251215 21:06:24.092536 25309 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba2ff0b3fab04c25bfe31f0957cdbc31. Latest consensus state: current_term: 2 leader_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2ff0b3fab04c25bfe31f0957cdbc31" member_type: VOTER } }
I20251215 21:06:24.092660 25309 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:24.093045 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [sys.catalog]: This master's current role is: LEADER
I20251215 21:06:24.093209 25315 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:06:24.094355 25315 catalog_manager.cc:679] Loaded metadata for table test-workload [id=db64f46791484d2f9164ead9fe284fa0]
I20251215 21:06:24.095067 25315 tablet_loader.cc:96] loaded metadata for tablet 9f979ea1cc6744ee840ec9d2f1cbccf1 (table test-workload [id=db64f46791484d2f9164ead9fe284fa0])
I20251215 21:06:24.095178 25315 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:06:24.095381 23617 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251215 21:06:24.095669 25315 catalog_manager.cc:1269] Loaded cluster ID: 92fa87506efc4403af87a9966379070b
I20251215 21:06:24.095715 25315 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:06:24.097051 25315 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:06:24.097539 25315 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31: Loaded TSK: 0
I20251215 21:06:24.097977 25315 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:06:24.099138 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251215 21:06:24.100935 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:24.101006 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.101029 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.101052 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:24.101315 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:24.101365 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:24.101384 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832784101384 us; error 0 us; skew 500 ppm
I20251215 21:06:24.102160 23617 webserver.cc:492] Webserver started at http://127.23.16.65:45993/ using document root <none> and password file <none>
I20251215 21:06:24.102303 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:24.102340 23617 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251215 21:06:24.103309 23617 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:24.104022 25332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.104308 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:24.104375 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root
uuid: "c57ae1126aa643a48dbd45219c6136f2"
format_stamp: "Formatted at 2025-12-15 21:06:23 on dist-test-slave-bslw"
I20251215 21:06:24.104449 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:24.122107 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:24.122514 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:24.122784 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:24.124346 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251215 21:06:24.124397 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251215 21:06:24.125097 23617 ts_tablet_manager.cc:616] Registered 1 tablets
I20251215 21:06:24.125134 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251215 21:06:24.125203 25339 tablet_bootstrap.cc:492] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Bootstrap starting.
I20251215 21:06:24.127944 25339 tablet_bootstrap.cc:492] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:06:24.128259 25339 tablet_bootstrap.cc:492] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Bootstrap complete.
I20251215 21:06:24.128386 25339 ts_tablet_manager.cc:1403] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251215 21:06:24.128784 25339 raft_consensus.cc:359] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.128916 25339 raft_consensus.cc:740] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57ae1126aa643a48dbd45219c6136f2, State: Initialized, Role: FOLLOWER
I20251215 21:06:24.129050 25339 consensus_queue.cc:260] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.129146 25339 raft_consensus.cc:399] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:06:24.129190 25339 raft_consensus.cc:493] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:06:24.129235 25339 raft_consensus.cc:3060] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:06:24.129586 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.65:42399
I20251215 21:06:24.129614 25397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.65:42399 every 8 connection(s)
I20251215 21:06:24.130134 25339 raft_consensus.cc:515] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
I20251215 21:06:24.130270 25339 leader_election.cc:304] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57ae1126aa643a48dbd45219c6136f2; no voters:
I20251215 21:06:24.130506 25339 leader_election.cc:290] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251215 21:06:24.130666 25401 raft_consensus.cc:2804] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 2 FOLLOWER]: Leader election won for term 2
I20251215 21:06:24.130856 25339 ts_tablet_manager.cc:1434] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20251215 21:06:24.130885 25401 raft_consensus.cc:697] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 2 LEADER]: Becoming Leader. State: Replica: c57ae1126aa643a48dbd45219c6136f2, State: Running, Role: LEADER
I20251215 21:06:24.130540 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251215 21:06:24.131071 25401 consensus_queue.cc:237] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } }
W20251215 21:06:24.132560 25407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.132772 25406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.133096 25410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:24.133148 23617 server_base.cc:1047] running on GCE node
I20251215 21:06:24.133291 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:24.133325 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:24.133345 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832784133345 us; error 0 us; skew 500 ppm
I20251215 21:06:24.134140 23617 webserver.cc:492] Webserver started at http://127.23.16.66:46319/ using document root <none> and password file <none>
I20251215 21:06:24.134290 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:24.134344 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:24.134411 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:24.134610 25398 heartbeater.cc:344] Connected to a master server at 127.23.16.126:37473
I20251215 21:06:24.134701 25398 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:24.134753 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-1-root/instance:
uuid: "707acf83e75e4e24a651066f07be9eef"
format_stamp: "Formatted at 2025-12-15 21:06:24 on dist-test-slave-bslw"
I20251215 21:06:24.134891 25398 heartbeater.cc:507] Master 127.23.16.126:37473 requested a full tablet report, sending...
I20251215 21:06:24.135630 25268 ts_manager.cc:194] Registered new tserver with Master: c57ae1126aa643a48dbd45219c6136f2 (127.23.16.65:42399)
I20251215 21:06:24.135959 25268 catalog_manager.cc:5654] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c57ae1126aa643a48dbd45219c6136f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57ae1126aa643a48dbd45219c6136f2" member_type: VOTER last_known_addr { host: "127.23.16.65" port: 34323 } health_report { overall_health: HEALTHY } } }
I20251215 21:06:24.136250 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251215 21:06:24.137133 25415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.137390 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:24.137485 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-1-root
uuid: "707acf83e75e4e24a651066f07be9eef"
format_stamp: "Formatted at 2025-12-15 21:06:24 on dist-test-slave-bslw"
I20251215 21:06:24.137576 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:24.138325 25268 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36876
I20251215 21:06:24.139004 25398 heartbeater.cc:499] Master 127.23.16.126:37473 was elected leader, sending a full tablet report...
I20251215 21:06:24.160949 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:24.161326 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:24.161620 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:24.162084 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:24.162122 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.162163 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:24.162191 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.166415 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.66:39941
I20251215 21:06:24.166461 25479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.66:39941 every 8 connection(s)
I20251215 21:06:24.167160 23617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:06:24.168825 25484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:06:24.168905 25483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:24.169045 23617 server_base.cc:1047] running on GCE node
W20251215 21:06:24.169107 25486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:06:24.169404 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251215 21:06:24.169453 23617 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251215 21:06:24.169473 23617 hybrid_clock.cc:648] HybridClock initialized: now 1765832784169473 us; error 0 us; skew 500 ppm
I20251215 21:06:24.170331 23617 webserver.cc:492] Webserver started at http://127.23.16.67:34989/ using document root <none> and password file <none>
I20251215 21:06:24.170480 23617 fs_manager.cc:362] Metadata directory not provided
I20251215 21:06:24.170527 23617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:06:24.170593 23617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:06:24.170945 23617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-2-root/instance:
uuid: "a56a60e8a1f44e669a216b46f0957f5a"
format_stamp: "Formatted at 2025-12-15 21:06:24 on dist-test-slave-bslw"
I20251215 21:06:24.171054 25480 heartbeater.cc:344] Connected to a master server at 127.23.16.126:37473
I20251215 21:06:24.171132 25480 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:24.171290 25480 heartbeater.cc:507] Master 127.23.16.126:37473 requested a full tablet report, sending...
I20251215 21:06:24.171854 25268 ts_manager.cc:194] Registered new tserver with Master: 707acf83e75e4e24a651066f07be9eef (127.23.16.66:39941)
I20251215 21:06:24.172436 23617 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:06:24.172487 25268 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36886
I20251215 21:06:24.173414 25491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.173619 23617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251215 21:06:24.173681 23617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-2-root
uuid: "a56a60e8a1f44e669a216b46f0957f5a"
format_stamp: "Formatted at 2025-12-15 21:06:24 on dist-test-slave-bslw"
I20251215 21:06:24.173744 23617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskL06uf9/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765832746896456-23617-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:06:24.190156 23617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:06:24.190572 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:06:24.190847 23617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:06:24.191291 23617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:06:24.191329 23617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.191372 23617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:06:24.191401 23617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:06:24.195874 23617 rpc_server.cc:307] RPC server started. Bound to: 127.23.16.67:33425
I20251215 21:06:24.195946 25554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.16.67:33425 every 8 connection(s)
I20251215 21:06:24.199908 25555 heartbeater.cc:344] Connected to a master server at 127.23.16.126:37473
I20251215 21:06:24.200006 25555 heartbeater.cc:461] Registering TS with master...
I20251215 21:06:24.200233 25555 heartbeater.cc:507] Master 127.23.16.126:37473 requested a full tablet report, sending...
I20251215 21:06:24.200829 25268 ts_manager.cc:194] Registered new tserver with Master: a56a60e8a1f44e669a216b46f0957f5a (127.23.16.67:33425)
I20251215 21:06:24.201346 25268 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36894
I20251215 21:06:24.201874 23617 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005636421s
I20251215 21:06:24.211261 23617 test_util.cc:276] Using random seed: -69866181
I20251215 21:06:24.218653 23617 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20251215 21:06:24.218789 23617 tablet_server.cc:178] TabletServer@127.23.16.65:0 shutting down...
I20251215 21:06:24.221639 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:24.221933 23617 tablet_replica.cc:333] stopping tablet replica
I20251215 21:06:24.222041 23617 raft_consensus.cc:2243] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 2 LEADER]: Raft consensus shutting down.
I20251215 21:06:24.222162 23617 raft_consensus.cc:2272] T 9f979ea1cc6744ee840ec9d2f1cbccf1 P c57ae1126aa643a48dbd45219c6136f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:24.234586 23617 tablet_server.cc:195] TabletServer@127.23.16.65:0 shutdown complete.
I20251215 21:06:24.236303 23617 tablet_server.cc:178] TabletServer@127.23.16.66:0 shutting down...
I20251215 21:06:24.239466 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:24.251052 23617 tablet_server.cc:195] TabletServer@127.23.16.66:0 shutdown complete.
I20251215 21:06:24.252830 23617 tablet_server.cc:178] TabletServer@127.23.16.67:0 shutting down...
I20251215 21:06:24.256006 23617 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251215 21:06:24.267640 23617 tablet_server.cc:195] TabletServer@127.23.16.67:0 shutdown complete.
I20251215 21:06:24.269379 23617 master.cc:561] Master@127.23.16.126:37473 shutting down...
I20251215 21:06:24.272399 23617 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 2 LEADER]: Raft consensus shutting down.
I20251215 21:06:24.272562 23617 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ba2ff0b3fab04c25bfe31f0957cdbc31 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251215 21:06:24.272702 23617 tablet_replica.cc:333] stopping tablet replica
I20251215 21:06:24.285355 23617 master.cc:583] Master@127.23.16.126:37473 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (403 ms)
[----------] 4 tests from TimestampAdvancementITest (37384 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (37385 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20251215 21:06:24.290861 23617 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5332: suppressed but not reported on 8 messages since previous log ~1 seconds ago
I20251215 21:06:24.290901 23617 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~2 seconds ago
I20251215 21:06:24.290917 23617 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20251215 21:06:24.290937 23617 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 1 messages since previous log ~0 seconds ago
I20251215 21:06:24.290956 23617 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 8 messages since previous log ~4 seconds ago
I20251215 21:06:24.291004 23617 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 574 messages since previous log ~37 seconds ago