Your question is Scaling Inference to Millions. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Discuss the challenges of scaling an inference service to handle millions of requests.
Explain how you would design the service for high throughput and predictable latency. Cover model serving, request routing, batching, caching, hardware selection, autoscaling, fault tolerance, deployment strategy, observability, and cost control. Discuss how you would identify and mitigate bottlenecks, handle traffic spikes, preserve model correctness, and detect training-serving skew or feature drift. Address how your design would change for CPU versus GPU models and for synchronous versus asynchronous inference.
