Attentive is evaluating a binary classification model that predicts whether a subscriber will click and convert after receiving an SMS campaign in the Attentive platform. The model is used to prioritize high-propensity subscribers for a limited-volume promotional send, but the growth team reports that recent campaigns are missing too many eventual converters.
| Metric | Validation Set | Previous Model | Change |
|---|---|---|---|
| Precision | 0.81 | 0.68 | +0.13 |
| Recall | 0.44 | 0.61 | -0.17 |
| F1 Score | 0.57 | 0.64 | -0.07 |
| AUC-ROC | 0.86 | 0.82 | +0.04 |
| Positive rate | 8.0% | 8.0% | 0.0% |
| Predicted positive rate | 4.3% | 7.2% | -2.9 pts |
The new model ranks users better overall by AUC-ROC, but at the current threshold it is much more conservative. As a result, Attentive sends fewer promotional messages and captures fewer total converters, despite higher precision.