Context

Ancestry has trained a gradient boosted classifier to rank record hints shown on an Ancestry member's family tree. The model predicts whether a user will accept a hint within 7 days. After launch, offline training performance looked excellent, but validation and holdout results were materially worse, and product partners are concerned the model may be overfitting.

Current Performance

The current model uses 140 features, including tree size, record type, historical hint acceptance behavior, session activity, and record-source metadata. Training used 4.2M labeled hints from Jan-Jun 2025; validation used Jul 2025; holdout test used Aug 2025.

The Problem

You need to determine whether this pattern indicates overfitting, underfitting, or a different evaluation issue, and recommend how Ancestry should validate and improve the model before wider rollout.

Requirements

1. Diagnose whether the model is overfitting or underfitting using the metrics above.
2. Explain which metric gaps matter most and why.
3. Propose a validation approach to confirm your diagnosis.
4. Recommend specific changes to improve generalization.
5. Discuss what additional error analysis you would run before shipping.

Constraints

• Hint ranking quality directly affects member trust and tree-building engagement.
• A false positive hint creates user frustration; a false negative hides a potentially valuable discovery.
• Retraining can be done weekly, but feature engineering changes take longer to productionize.