Evaluate Beyond Accuracy for Leads | Dataford Interview Questions

Your question is Evaluate Beyond Accuracy for Leads. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Context

Capital Group is evaluating a binary classification model in Capital Ideas Pro that predicts whether a financial advisor will respond to an outreach campaign. The team initially celebrated because the model achieved high accuracy, but campaign managers noticed that many likely responders were still being missed.

Current Performance

The model was evaluated on a holdout set of 10,000 advisors, where 1,000 advisors actually responded and 9,000 did not.

Metric	Value
Accuracy	0.91
Precision	0.67
Recall	0.40
F1 Score	0.50
AUC-ROC	0.84
Log Loss	0.29
Threshold	0.50

Confusion matrix counts:

	Predicted Respond	Predicted No Respond
Actual Respond	400	600
Actual No Respond	200	8,800

The Problem

Accuracy looks strong at 91%, but the model only identifies 40% of actual responders. The distribution is imbalanced, so relying on accuracy alone may lead to poor business decisions about advisor targeting.

Requirements

Explain why accuracy is an incomplete metric in this case.
Interpret precision, recall, F1, AUC-ROC, and log loss using the numbers above.
Describe what the confusion matrix says about business impact.
Recommend which metrics Capital Group should prioritize for this use case and why.
Suggest specific ways to improve evaluation and model performance.

Constraints

Outreach budget allows contacting only 1,200 advisors per campaign.
Missing a likely responder is more costly than contacting a non-responder.
Relationship managers want interpretable model evaluation, not just a single headline metric.