Context

VectorServe runs a distributed recommendation inference service across GPU nodes. The team recently upgraded the serving stack from 8 to 64 GPUs, but product latency complaints increased and infrastructure cost rose faster than throughput. Leadership wants a clear evaluation of throughput, latency, utilization, strong scaling, weak scaling, and the gap between observed speedup and Amdahl’s law.

Current Performance

The Problem

The system adds hardware, but efficiency falls sharply at larger cluster sizes. You need to determine which metrics best capture this behavior and whether the bottleneck is parallel overhead, communication, or poor resource utilization.

Requirements

1. Define the primary metrics you would use for throughput, latency, utilization, strong scaling, weak scaling, and Amdahl’s law.
2. Interpret the results above and quantify scaling efficiency at each cluster size.
3. Estimate the implied serial fraction from the observed speedup at 64 GPUs.
4. Identify the most likely bottlenecks and the tradeoffs between throughput and tail latency.
5. Recommend concrete measurement and optimization steps.

Constraints

• P95 latency must stay below 160 ms for production traffic.
• GPU fleet cost doubles when moving from 32 to 64 GPUs.
• The service must support both fixed-load and proportional-load scaling tests.