Online vs Batch Model Serving

MediumSystem Design00:00

Practice interviewer

In session

5 left

00:00

Your question is Online vs Batch Model Serving. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

Scenario

You are building an ML powered ranking system for a consumer product feed. Some predictions can be precomputed ahead of time, while others depend on fresh user context and must be computed at request time. You need to decide how to split inference between batch and online paths.

Question

What are the trade-offs between online and batch serving for ML models?

What this tests

Choosing between batch and online inference by feature volatility
Designing a retrieval → ranking → re-ranking stack
Using a feature store to reduce training-serving skew
Handling feature drift, fallback paths, and monitoring