Design a Fair Cross-Hardware Benchmark

Context

NimbusAI serves a text-generation API and is evaluating a new 7B-parameter model for production. The team tested the model on three hardware platforms, but leadership does not trust the results because throughput and latency vary materially by device, software stack, and batch settings.

Current Performance

The Problem

The model quality metrics are similar, but the benchmark is not reproducible or obviously fair: each platform used different batch sizes, kernels, quantization settings, and warm-up durations. Some runs excluded tokenization time; others reported steady-state throughput only. Your task is to redesign the benchmark so results can be compared credibly across hardware platforms.

Requirements

1. Identify which parts of the current benchmark make the comparison unfair or non-reproducible.
2. Define a benchmark protocol that standardizes workload, software, and measurement methodology.
3. Recommend which quality and system metrics should be reported together and why.
4. Explain how to handle unavoidable hardware-specific optimizations without biasing the comparison.
5. Propose a validation plan to ensure repeated runs produce stable results.

Constraints

• NimbusAI must support both online inference (P95 latency < 150 ms) and batch summarization jobs.
• The benchmark must be runnable by external partners with limited access to proprietary tooling.
• Engineering can afford at most 5 repeated runs per platform per benchmark configuration.