At SparkFit, a subscription fitness app, the growth team uses a binary classifier to predict which free users are likely to convert to paid within 30 days so they can target discount offers. The model reports very high overall accuracy, but campaign lift has been disappointing and the paid marketing budget is being wasted on the wrong users.
| Metric | Value |
|---|---|
| Accuracy | 96.2% |
| Precision | 28.0% |
| Recall | 11.7% |
| F1 Score | 16.5% |
| AUC-ROC | 0.74 |
| Positive rate in validation set | 4.0% |
| Users scored in monthly campaign | 500,000 |
| Users predicted positive at current threshold | 8,300 |
| Actual converters in validation set | 4,000 / 100,000 |
Leadership sees 96.2% accuracy and assumes the model is strong. However, only a small fraction of actual future converters are being identified, which limits incremental revenue in a growth use case where missing high-intent users is expensive.