Detect Leakage in Feature Engineering

Context

ShopLens built a binary classification model to predict whether a customer will make a repeat purchase within 30 days of their first order. A LightGBM model looked excellent in offline validation, but performance dropped sharply after deployment. The team suspects data leakage introduced during feature engineering.

Current Performance

The feature set includes customer tenure, average basket size, email engagement, support contacts, and rolling 30-day order aggregates. During review, the team found that some aggregates may have been computed using data extending beyond the prediction timestamp.

The Problem

You need to determine whether the performance gap is caused by leakage, identify which features or validation steps are most suspicious, and recommend how to redesign the feature engineering and evaluation process.

Requirements

1. Diagnose the most likely leakage mechanisms given the metric pattern.
2. Identify which features and validation choices should be audited first.
3. Explain how you would prove leakage using targeted experiments.
4. Recommend changes to feature generation, train/validation splitting, and monitoring.
5. Discuss how you would rebuild trust in the model before redeployment.

Constraints

• Marketing uses the model weekly for retention campaigns.
• False positives waste outreach budget; false negatives miss high-value repeat buyers.
• Full retraining and backfill must be completed within 10 days.