On ai and Feature stores

On ai and Feature stores

How are feature stores used in GenAI?

Search for: How are feature stores used in GenAI?

  1. X - Various input columns
  2. Y - One or many Outputs given X
  3. X is called the feature set (The list of columns)
  4. Y is sometimes called the labels or outputs
  5. You train a model by passing (X,Y)
  6. The model then learns and is represented by a set of (parameters) which can work without the X and Y.
  1. it is a database of features that can participate in variety of experiments satisfying the "X".
  2. The word "Entity" refers to the table that keeps the columns of a given set of "X"
  3. There can be many such feature tables, each table having many features
  4. Then during training one can take a subset of these to train their models
  5. It also has a role during inference (or real time APIs)
  1. Can use any ETL including Spark
  2. You can store them S3, HDFS or a warehouse that sure is suitable for batch access
  3. Or a store that is optimized for both real time and batch like Azure ADX for instance.
  1. Given something like a user id,
  2. You can then ask the feature store to get a subset of the features
  3. To feed a model that is trained on those features to get back an answer
  1. metadata that describes all features for discovery and cataloging
  2. Reconciliation service that guarantees data consistency as it gets duplicated for any reason during source copies or during real time vs batch copies

Medium: Deeper article by the same folks of the above video

Feature stores for ML: Medium collection of articles

  1. At the end of the day it is a set of tables and columns
  2. Albeit Specifically designed with cleaned up data suitable for ML training and ML prediction (inference)
  3. Need to allow for ETL ingestion and consumption
  4. Need to allow API access

Michaelangelo, Uber:Search On Web

Feast:Search On Web

HopsWorks:Search On Web

Databricks feature store:Search On Web

Google Vertex:Search On Web

Feature stores and LLMs

Search for: Feature stores and LLMs

LangChain: Feature stores and LLMs

Tecton:Search On Web

FeatureForm:Search On Web

Feast:Search On Web

Micrsoft's take on feature store

Setting up a feast feature store

RAGs and LLMs: from Hops Feature store guys

What is a feature store: A linked in Guide