Evaluate Customer Churn Classifier Metrics | Dataford Interview Questions

Context

StreamWave has built a binary classification model to predict whether a subscriber will churn in the next 30 days so the retention team can send targeted offers. The current logistic regression model performs well on overall accuracy, but the retention team says too many likely churners are being missed.

Current Performance

Metric	Validation Value
Accuracy	0.91
Precision	0.64
Recall	0.38
F1 Score	0.48
AUC-ROC	0.79
Log Loss	0.31
Positive Class Rate	0.12

The Problem

The VP of Growth is concerned that the model looks strong by accuracy alone, yet only a minority of actual churners are being identified. The retention budget is limited, so unnecessary outreach also has a cost.

Requirements

Explain which classification metrics you would prioritize for this use case and why.
Interpret the current metric values together rather than in isolation.
Use the confusion matrix to describe the model's main failure mode.
Recommend whether StreamWave should adjust the decision threshold.
Propose specific next steps to improve model evaluation and model performance.

Constraints

Monthly retention campaign capacity: 8,000 users
Cost of unnecessary outreach: $2 per user contacted
Estimated lost revenue from a missed churner: $65
The model must remain interpretable for the marketing team

Problem

Context

Current Performance

Metric	Validation Value
Accuracy	0.91
Precision	0.64
Recall	0.38
F1 Score	0.48
AUC-ROC	0.79
Log Loss	0.31
Positive Class Rate	0.12

The Problem

Requirements

Explain which classification metrics you would prioritize for this use case and why.
Interpret the current metric values together rather than in isolation.
Use the confusion matrix to describe the model's main failure mode.
Recommend whether StreamWave should adjust the decision threshold.
Propose specific next steps to improve model evaluation and model performance.

Constraints

Monthly retention campaign capacity: 8,000 users
Cost of unnecessary outreach: $2 per user contacted
Estimated lost revenue from a missed churner: $65
The model must remain interpretable for the marketing team

Problem

Context

Current Performance

Metric	Validation Value
Accuracy	0.91
Precision	0.64
Recall	0.38
F1 Score	0.48
AUC-ROC	0.79
Log Loss	0.31
Positive Class Rate	0.12

The Problem

Requirements

Explain which classification metrics you would prioritize for this use case and why.
Interpret the current metric values together rather than in isolation.
Use the confusion matrix to describe the model's main failure mode.
Recommend whether StreamWave should adjust the decision threshold.
Propose specific next steps to improve model evaluation and model performance.

Constraints

Monthly retention campaign capacity: 8,000 users
Cost of unnecessary outreach: $2 per user contacted
Estimated lost revenue from a missed churner: $65
The model must remain interpretable for the marketing team

Problem

Context

Current Performance

Metric	Validation Value
Accuracy	0.91
Precision	0.64
Recall	0.38
F1 Score	0.48
AUC-ROC	0.79
Log Loss	0.31
Positive Class Rate	0.12

The Problem

Requirements

Explain which classification metrics you would prioritize for this use case and why.
Interpret the current metric values together rather than in isolation.
Use the confusion matrix to describe the model's main failure mode.
Recommend whether StreamWave should adjust the decision threshold.
Propose specific next steps to improve model evaluation and model performance.

Constraints

Monthly retention campaign capacity: 8,000 users
Cost of unnecessary outreach: $2 per user contacted
Estimated lost revenue from a missed churner: $65
The model must remain interpretable for the marketing team