Your question is Scalable LLM Inference at 1000 RPS. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Design a scalable inference service for a LLM that must handle 1000 requests per second with strict latency requirements.
Clarify the latency target and request characteristics, then describe the serving architecture, capacity planning, batching strategy, model optimization, observability, and failure handling. Address online inference, autoscaling, cost, quality evaluation, and training-serving consistency without assuming a specific cloud provider or model.
