Your question is Choosing Load Balancing Strategies. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Discuss the trade-offs when choosing between different load balancing strategies.
For an ML-serving system, explain how you would compare round robin, weighted routing, least-connections, consistent hashing, and model-aware or latency-aware routing. Address uneven request costs, autoscaling, cache locality, overloaded or unhealthy replicas, model versions, failover, observability, and tail latency. Treat this as a practical design discussion: state assumptions, recommend an approach, and explain how you would validate it.
