Evaluate Customer Churn Model Performance | Dataford Interview Questions

Context

StreamWave uses a binary classification model to predict which paid subscribers are likely to cancel in the next 30 days so the CRM team can send retention offers. The current model was deployed 2 months ago, but leadership is concerned that campaign ROI is below target despite seemingly strong overall accuracy.

Current Performance

Metric	Validation Set	Current Production	Change
Accuracy	0.91	0.89	-0.02
Precision	0.62	0.48	-0.14
Recall	0.71	0.36	-0.35
F1 Score	0.66	0.41	-0.25
AUC-ROC	0.84	0.79	-0.05
Churn Rate	0.12	0.10	-0.02
Users flagged / month	18,000	9,500	-8,500
Actual churners / month	12,000	10,000	-2,000

The Problem

The model still shows high accuracy, but it is identifying far fewer churners than expected. As a result, many at-risk users are not receiving retention offers, and the retention team wants to know whether the issue is threshold choice, class imbalance, drift, or segment-specific underperformance.

Requirements

Interpret the current metrics and explain why accuracy may be misleading here.
Diagnose the most likely reasons recall dropped from 0.71 to 0.36.
Use the confusion matrix and business constraints to assess whether the current threshold is appropriate.
Identify which additional validation or error analysis you would run.
Recommend specific model or process improvements to improve performance.

Constraints

Retention budget supports at most 15,000 offers per month.
Each offer costs $8.
Saving a true churner is worth $95 in expected annual gross profit.
Product and pricing changed 6 weeks after deployment.

Problem

Context

Current Performance

Metric	Validation Set	Current Production	Change
Accuracy	0.91	0.89	-0.02
Precision	0.62	0.48	-0.14
Recall	0.71	0.36	-0.35
F1 Score	0.66	0.41	-0.25
AUC-ROC	0.84	0.79	-0.05
Churn Rate	0.12	0.10	-0.02
Users flagged / month	18,000	9,500	-8,500
Actual churners / month	12,000	10,000	-2,000

The Problem

Requirements

Interpret the current metrics and explain why accuracy may be misleading here.
Diagnose the most likely reasons recall dropped from 0.71 to 0.36.
Use the confusion matrix and business constraints to assess whether the current threshold is appropriate.
Identify which additional validation or error analysis you would run.
Recommend specific model or process improvements to improve performance.

Constraints

Retention budget supports at most 15,000 offers per month.
Each offer costs $8.
Saving a true churner is worth $95 in expected annual gross profit.
Product and pricing changed 6 weeks after deployment.

Problem

Context

Current Performance

Metric	Validation Set	Current Production	Change
Accuracy	0.91	0.89	-0.02
Precision	0.62	0.48	-0.14
Recall	0.71	0.36	-0.35
F1 Score	0.66	0.41	-0.25
AUC-ROC	0.84	0.79	-0.05
Churn Rate	0.12	0.10	-0.02
Users flagged / month	18,000	9,500	-8,500
Actual churners / month	12,000	10,000	-2,000

The Problem

Requirements

Interpret the current metrics and explain why accuracy may be misleading here.
Diagnose the most likely reasons recall dropped from 0.71 to 0.36.
Use the confusion matrix and business constraints to assess whether the current threshold is appropriate.
Identify which additional validation or error analysis you would run.
Recommend specific model or process improvements to improve performance.

Constraints

Retention budget supports at most 15,000 offers per month.
Each offer costs $8.
Saving a true churner is worth $95 in expected annual gross profit.
Product and pricing changed 6 weeks after deployment.

Problem

Context

Current Performance

Metric	Validation Set	Current Production	Change
Accuracy	0.91	0.89	-0.02
Precision	0.62	0.48	-0.14
Recall	0.71	0.36	-0.35
F1 Score	0.66	0.41	-0.25
AUC-ROC	0.84	0.79	-0.05
Churn Rate	0.12	0.10	-0.02
Users flagged / month	18,000	9,500	-8,500
Actual churners / month	12,000	10,000	-2,000

The Problem

Requirements

Interpret the current metrics and explain why accuracy may be misleading here.
Diagnose the most likely reasons recall dropped from 0.71 to 0.36.
Use the confusion matrix and business constraints to assess whether the current threshold is appropriate.
Identify which additional validation or error analysis you would run.
Recommend specific model or process improvements to improve performance.

Constraints

Retention budget supports at most 15,000 offers per month.
Each offer costs $8.
Saving a true churner is worth $95 in expected annual gross profit.
Product and pricing changed 6 weeks after deployment.