Business Context

ArenaForge runs a competitive multiplayer game with 12M monthly players. The gameplay analytics team built a model on player telemetry to predict whether a player will churn in the next 7 days, but offline results look much better than live performance, suggesting overfitting.

Dataset

You are given a supervised learning dataset built from player-session telemetry aggregated to the player-day level.

• Rows: 420K player-days across 9 months
• Target: Binary label indicating churn within the next 7 days
• Class balance: 18% positive, 82% negative
• Missing data: 6% missing in network/device fields, 9% missing in economy features for non-spenders

Success Criteria

A strong solution should clearly diagnose whether the model is overfitting, quantify the train/validation/test gap, identify likely causes such as leakage or excessive complexity, and recommend fixes that improve generalization. A good target is reducing the train-test AUC gap to < 0.03 while maintaining test ROC-AUC e 0.78.

Constraints

• Predictions are generated daily in batch for up to 2M active players
• The analytics team needs interpretable diagnostics, not just a better model
• Avoid temporal leakage: future behavior cannot influence current predictions

Deliverables

1. Define a validation strategy to detect overfitting on telemetry data.
2. Train at least one baseline model and one higher-capacity model.
3. Compare train, validation, and test performance using appropriate metrics.
4. Diagnose likely overfitting causes and propose mitigation steps.
5. Provide code to reproduce preprocessing, training, and evaluation.