Context

ShopLens runs semantic product retrieval for a marketplace with 180M item embeddings and 12K queries per second at peak. The team recently moved ANN search from 1 GPU to 4 GPUs to preserve recall at larger corpus size, but latency is now above the serving SLO even though retrieval quality remains strong.

Current Performance

The Problem

The retrieval team believes the index scaling strategy improved recall, but the multi-GPU deployment introduced a latency bottleneck somewhere in query fan-out, candidate merge, memory movement, or load imbalance. You need to diagnose the most likely bottlenecks and recommend the first optimizations to try.

Requirements

1. Interpret what the metric pattern suggests about system bottlenecks.
2. Identify the most likely latency contributors in a multi-GPU ANN stack.
3. Prioritize the first 4-5 optimizations you would test.
4. Explain what tradeoffs you would monitor so recall does not regress.
5. Propose a validation plan to confirm the true root cause.

Constraints

• Recall@100 cannot fall below 0.960.
• No additional GPUs this quarter.
• End-to-end P95 latency must reach 100 ms within 3 weeks.
• Rebuilding the full index takes 36 hours, so experiments must be staged carefully.