Validate Generalization Before Deployment

Context

ShopLens is building a binary classification model to predict whether a user will purchase within 7 days after viewing a product. A gradient boosting model was trained on 1.2M sessions from Jan-Jun 2024 and is being considered for deployment to drive retargeting spend. The team is concerned that strong validation results may not hold on truly unseen traffic.

Current Performance

The Problem

Performance degrades consistently from train to cross-validation to holdout to the most recent out-of-time test. Marketing wants to launch the model next month, but the data science lead wants evidence that it will generalize to unseen data and remain reliable under changing traffic patterns.

Requirements

1. Interpret whether the model is overfitting, suffering from data drift, or both.
2. Explain which validation scheme best estimates real-world performance and why.
3. Identify what additional checks you would run before deployment.
4. Recommend concrete steps to improve generalization and reduce deployment risk.
5. Discuss which metric should drive the launch decision given retargeting budget constraints.

Constraints

• Monthly retargeting budget is fixed at $400K.
• False positives waste ad spend; false negatives miss revenue opportunities.
• Retraining can happen at most once per month.
• The model must score traffic in near real time.