Ancestra, a genealogy platform, uses a ranking model to recommend personalized family history experiences such as photo collections, census records, story prompts, and DNA-relative connections. The model was launched to increase weekly engagement, but product leaders now see strong click behavior with weaker downstream completion and subscription conversion.
| Metric | Offline Validation | Online Last 30 Days | Target |
|---|---|---|---|
| Precision@5 | 0.41 | 0.38 | 0.45 |
| Recall@5 | 0.29 | 0.24 | 0.30 |
| F1@5 | 0.34 | 0.29 | 0.36 |
| AUC-ROC (click prediction) | 0.81 | 0.78 | 0.82 |
| Log Loss | 0.49 | 0.57 | <0.50 |
| Calibration error | 0.06 | 0.14 | <0.08 |
| CTR on recommended experiences | 7.8% | 8.4% | 8.0% |
| Completion rate after click | 34% | 22% | 30% |
| Subscription conversion from recommenders | 3.1% | 1.9% | 2.8% |
The recommendation engine appears optimized for clicks, but many clicked recommendations do not lead to meaningful family history engagement. Leadership wants to know whether the model is poorly calibrated, using the wrong objective, or underperforming for key user segments such as new users and older family-tree builders.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
