Assess Cross-Validation for Churn Model

Context

StreamBox built a logistic regression model to predict 30-day subscription churn so the CRM team can target retention offers. On a single 20% holdout split, the model looked strong, but leadership is concerned the result may be overly optimistic because performance varies by data split.

Current Performance

Metric	Single Holdout	5-Fold Cross-Validation Mean	Fold Std Dev
Accuracy	0.91	0.84	0.05
Precision	0.72	0.61	0.08
Recall	0.68	0.49	0.10
F1 Score	0.70	0.54	0.09
AUC-ROC	0.88	0.79	0.06
Positive class rate	0.18	0.18	0.01

Fold 1 F1	-	0.66	-
Fold 2 F1	-	0.58	-
Fold 3 F1	-	0.52	-
Fold 4 F1	-	0.47	-
Fold 5 F1	-	0.45	-

The Problem

The team needs to explain why cross-validation matters here and decide whether the model is reliable enough to launch. The gap between holdout and cross-validation suggests the single split may not reflect true generalization performance.

Requirements

Explain what cross-validation is and why it is useful in model evaluation.
Interpret the gap between the single holdout metrics and the cross-validation averages.
Assess whether the model appears stable across folds.
Identify likely risks of relying only on the holdout result.
Recommend how the team should evaluate and improve the model before deployment.

Constraints

Retention budget allows offers to only 8% of active subscribers.
Missing a true churner is estimated to cost $42 in lost margin.
Unnecessary offers cost $6 per customer.

Context

Current Performance

Metric	Single Holdout	5-Fold Cross-Validation Mean	Fold Std Dev
Accuracy	0.91	0.84	0.05
Precision	0.72	0.61	0.08
Recall	0.68	0.49	0.10
F1 Score	0.70	0.54	0.09
AUC-ROC	0.88	0.79	0.06
Positive class rate	0.18	0.18	0.01

Fold 1 F1	-	0.66	-
Fold 2 F1	-	0.58	-
Fold 3 F1	-	0.52	-
Fold 4 F1	-	0.47	-
Fold 5 F1	-	0.45	-

The Problem

Requirements

Explain what cross-validation is and why it is useful in model evaluation.
Interpret the gap between the single holdout metrics and the cross-validation averages.
Assess whether the model appears stable across folds.
Identify likely risks of relying only on the holdout result.
Recommend how the team should evaluate and improve the model before deployment.

Constraints

Retention budget allows offers to only 8% of active subscribers.
Missing a true churner is estimated to cost $42 in lost margin.
Unnecessary offers cost $6 per customer.

Context

Current Performance

Metric	Single Holdout	5-Fold Cross-Validation Mean	Fold Std Dev
Accuracy	0.91	0.84	0.05
Precision	0.72	0.61	0.08
Recall	0.68	0.49	0.10
F1 Score	0.70	0.54	0.09
AUC-ROC	0.88	0.79	0.06
Positive class rate	0.18	0.18	0.01

Fold 1 F1	-	0.66	-
Fold 2 F1	-	0.58	-
Fold 3 F1	-	0.52	-
Fold 4 F1	-	0.47	-
Fold 5 F1	-	0.45	-

The Problem

Requirements

Explain what cross-validation is and why it is useful in model evaluation.
Interpret the gap between the single holdout metrics and the cross-validation averages.
Assess whether the model appears stable across folds.
Identify likely risks of relying only on the holdout result.
Recommend how the team should evaluate and improve the model before deployment.

Constraints

Retention budget allows offers to only 8% of active subscribers.
Missing a true churner is estimated to cost $42 in lost margin.
Unnecessary offers cost $6 per customer.

Context

Current Performance

Metric	Single Holdout	5-Fold Cross-Validation Mean	Fold Std Dev
Accuracy	0.91	0.84	0.05
Precision	0.72	0.61	0.08
Recall	0.68	0.49	0.10
F1 Score	0.70	0.54	0.09
AUC-ROC	0.88	0.79	0.06
Positive class rate	0.18	0.18	0.01

Fold 1 F1	-	0.66	-
Fold 2 F1	-	0.58	-
Fold 3 F1	-	0.52	-
Fold 4 F1	-	0.47	-
Fold 5 F1	-	0.45	-

The Problem

Requirements

Explain what cross-validation is and why it is useful in model evaluation.
Interpret the gap between the single holdout metrics and the cross-validation averages.
Assess whether the model appears stable across folds.
Identify likely risks of relying only on the holdout result.
Recommend how the team should evaluate and improve the model before deployment.

Constraints

Retention budget allows offers to only 8% of active subscribers.
Missing a true churner is estimated to cost $42 in lost margin.
Unnecessary offers cost $6 per customer.

Interview Guides

Context

Current Performance

The Problem

Requirements

Constraints

Assess Cross-Validation for Churn Model

Context

Current Performance

The Problem

Requirements

Constraints

Your Answer

Assess Cross-Validation for Churn Model

Context

Current Performance

The Problem

Requirements

Constraints

Assess Cross-Validation for Churn Model

Context

Current Performance

The Problem

Requirements

Constraints

Your Answer