Diagnose Bias-Variance in Churn Models

Business Context

Royal Cyber wants to improve retention for its managed commerce and digital transformation clients by predicting which customer accounts are likely to churn in the next 90 days. The current model performs inconsistently across training and validation data, and your task is to identify whether the issue is high bias, high variance, or both, then improve the model without making it too slow or opaque for the customer success team.

Dataset

You are given an account-level dataset built from Royal Cyber CRM, support, billing, and product usage systems.

• Size: 92,000 account-month records from the last 24 months, 52 features
• Target: Binary label indicating whether the account churns within the next 90 days
• Class balance: 11.4% churn, 88.6% retained
• Missing data: 12% missing in support fields, 6% missing in usage fields for newly onboarded accounts

Success Criteria

A strong solution should clearly diagnose bias vs. variance using train/validation behavior, improve generalization, and achieve ROC-AUC >= 0.84, PR-AUC >= 0.46, and F1 >= 0.58 on the held-out test set.

Constraints

• Predictions run nightly in batch for ~40K active accounts
• The customer success team needs understandable drivers behind risk scores
• Retraining should fit within a 2-hour Royal Cyber data science workflow

Deliverables

1. Build a baseline model and quantify train vs. validation performance.
2. Diagnose whether the baseline suffers from high bias, high variance, or both.
3. Propose and implement fixes using model complexity, regularization, feature engineering, or data strategy.
4. Compare at least two model families and justify the final choice.
5. Report final metrics, threshold choice, and top features influencing churn risk.