Design a Real-Time ML Feature Store

Scenario

You are designing a feature store for a large consumer marketplace that powers multiple ML systems, including search retrieval, ranking, fraud detection, and personalized recommendations. Today, each team computes features separately, causing inconsistent definitions between training and serving, slow experimentation, and frequent training-serving skew. You have been asked to build a shared feature platform that supports both offline training datasets and low-latency online inference. The goal is to make feature computation reusable, fresh, and reliable across models while reducing leakage, drift, and operational overhead.

Scale

Question

How would you design this feature store end to end so that it supports both model training and online serving at this scale? Explain the architecture, data flow, consistency strategy, evaluation approach, and how you would handle drift, skew, backfills, and failures.