Your question is Real-Time LLM Serving. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Given a requirement for real-time inference, how would you approach system design for LLM serving to optimize for throughput and latency?
Discuss how you would clarify workload requirements, select hardware and inference optimizations, design online serving, and handle capacity, failures, and observability. Include tradeoffs between batching, model quality, cost, throughput, and tail latency, plus how you would evaluate the system before and after deployment.