Validate Offline Metrics Against Business Value

Context

StreamCart is building a propensity model to predict which users will purchase a premium subscription within 7 days after seeing an in-app upsell. The team has strong offline model metrics, but product leadership is not convinced those metrics translate into real business value or justify an online launch.

Current Performance

The Problem

The model looks better offline, but the business decision is whether targeting the top-scored users will create enough incremental subscription revenue to offset notification cost and user fatigue. You need to design a validation approach that connects offline metrics to expected business impact before running a full A/B test.

Requirements

1. Explain which offline metrics are most relevant for validating model value and why.
2. Propose a framework to estimate expected business impact from the holdout results.
3. Identify what additional offline analyses you would run before launch.
4. Recommend how to choose a targeting threshold or top-K policy.
5. Describe the main risks of relying only on offline metrics.

Constraints

• Premium subscription revenue: $18 per conversion
• Notification cost: $0.04 per message sent
• Product team will target at most 500,000 users per week
• Excessive messaging can reduce 30-day retention