Interpret AUC-ROC for Marketing Model

Context

BrightCart uses a binary classification model to predict which users are likely to respond to a paid email promotion for a premium subscription upgrade. The marketing team is encouraged by the model's AUC-ROC, but campaign ROI has been inconsistent because the final audience is selected using a fixed score threshold.

Current Performance

The Problem

The VP of Marketing asks whether an AUC-ROC of 0.84 means the model is "good enough" for rollout across all campaigns. The team wants to understand what this score actually says about ranking quality, what it does not guarantee, and whether threshold or calibration issues could still hurt business outcomes.

Requirements

1. Interpret the AUC-ROC score of 0.84 in plain business terms.
2. Explain how AUC-ROC should be read alongside precision, recall, and base conversion rate.
3. Identify why a strong AUC-ROC may still lead to weak campaign performance at a chosen threshold.
4. Recommend how to evaluate threshold selection for profit, not just discrimination.
5. Suggest next steps to validate whether the model is ready for production.

Constraints

• Marketing can contact at most 15% of the user base per campaign.
• Each email costs $0.04 to send.
• A successful upgrade yields $12 in expected gross margin.
• Over-targeting low-propensity users reduces brand engagement over time.