Context

Attentive is evaluating a binary classification model that predicts whether a subscriber will click and convert after receiving an SMS campaign in the Attentive platform. The model is used to prioritize high-propensity subscribers for a limited-volume promotional send, but the growth team reports that recent campaigns are missing too many eventual converters.

Current Performance

The Problem

The new model ranks users better overall by AUC-ROC, but at the current threshold it is much more conservative. As a result, Attentive sends fewer promotional messages and captures fewer total converters, despite higher precision.

Requirements

1. Explain what the precision, recall, and AUC-ROC values imply for this use case.
2. Diagnose whether the model is actually better than the previous model, and under what definition of success.
3. Use the confusion matrix and business constraints to argue whether the threshold should change.
4. Recommend which metric should be primary for model selection in this setting and why.
5. Propose 3-4 concrete next steps to improve evaluation and model performance.

Constraints

• SMS volume is capped due to deliverability and budget limits.
• False positives waste message spend and may increase unsubscribe risk.
• False negatives miss potential revenue from high-intent subscribers.
• The marketing team can tolerate a modest precision drop if total conversions increase.