Context

Capital Group is evaluating a binary classification model in Capital Ideas Pro that predicts whether a financial advisor will respond to an outreach campaign. The team initially celebrated because the model achieved high accuracy, but campaign managers noticed that many likely responders were still being missed.

Current Performance

The model was evaluated on a holdout set of 10,000 advisors, where 1,000 advisors actually responded and 9,000 did not.

Confusion matrix counts:

The Problem

Accuracy looks strong at 91%, but the model only identifies 40% of actual responders. The distribution is imbalanced, so relying on accuracy alone may lead to poor business decisions about advisor targeting.

Requirements

1. Explain why accuracy is an incomplete metric in this case.
2. Interpret precision, recall, F1, AUC-ROC, and log loss using the numbers above.
3. Describe what the confusion matrix says about business impact.
4. Recommend which metrics Capital Group should prioritize for this use case and why.
5. Suggest specific ways to improve evaluation and model performance.

Constraints

• Outreach budget allows contacting only 1,200 advisors per campaign.
• Missing a likely responder is more costly than contacting a non-responder.
• Relationship managers want interpretable model evaluation, not just a single headline metric.