StreamMobile uses a lift model to target retention offers to subscribers at risk of churn. The model predicts incremental conversion from sending a growth offer (3 free months of premium) rather than just propensity to accept the offer.
A recent offline evaluation looks promising on ranking metrics, but the growth team is unsure whether the model is actually useful for targeting because offer cost is high and only 20% of eligible users can be contacted each month.
| Metric | Current Model | Random Targeting | Existing Heuristic |
|---|---|---|---|
| Qini coefficient | 0.118 | 0.000 | 0.041 |
| AUUC | 0.072 | 0.000 | 0.025 |
| Top 10% incremental conversion lift | 5.8 pp | 0.0 pp | 2.1 pp |
| Top 20% incremental conversion lift | 4.1 pp | 0.0 pp | 1.6 pp |
| Average treatment effect (ATE) | 1.2 pp | 1.2 pp | 1.2 pp |
| Offer acceptance rate in targeted top 20% | 18.4% | 14.3% | 15.1% |
| 90-day retention in targeted top 20% | 74.2% | 71.0% | 71.8% |
| Estimated offer cost per user | $9.00 | $9.00 | $9.00 |
| Estimated 90-day gross profit per retained user | $42.00 | $42.00 | $42.00 |
The VP of Growth wants to know whether the model should replace the current rule-based targeting strategy. You need to determine whether the model creates enough incremental value, whether the offline metrics are trustworthy, and how to validate the model before a full rollout.