Design Offline Validation for Ranking Model

Context

ShopNow wants to replace its current product recommendation ranker on the homepage with a new gradient-boosted model that predicts 7-day purchase probability. Leadership wants an offline validation framework that can credibly estimate business value before running an online experiment. The challenge is that the model scores items shown in historical logs, where exposure was determined by the current production ranker.

Current Performance

The Problem

The candidate model looks better on ranking metrics, but its probabilities are poorly calibrated and evaluation is based only on logged impressions from the old system. You need to design an offline validation framework that can demonstrate likely value, identify risks, and define what evidence is strong enough to justify an A/B test.

Requirements

1. Define the offline datasets, splits, and holdout strategy you would use.
2. Explain which metrics you would trust most for proving value and why.
3. Address selection bias from historical exposure under the production ranker.
4. Propose how to validate calibration, thresholding, and segment-level performance.
5. Recommend clear go/no-go criteria for launching an online test.

Constraints

• 120M logged impressions from the last 90 days
• Strong seasonality around weekends and promotions
• Only 8% of catalog items receive meaningful exposure
• Online experiment capacity is limited to one test this quarter