Evaluate Family History Recommendations

Context

Ancestra, a genealogy platform, uses a ranking model to recommend personalized family history experiences such as photo collections, census records, story prompts, and DNA-relative connections. The model was launched to increase weekly engagement, but product leaders now see strong click behavior with weaker downstream completion and subscription conversion.

Current Performance

Metric	Offline Validation	Online Last 30 Days	Target
Precision@5	0.41	0.38	0.45
Recall@5	0.29	0.24	0.30
F1@5	0.34	0.29	0.36
AUC-ROC (click prediction)	0.81	0.78	0.82
Log Loss	0.49	0.57	<0.50
Calibration error	0.06	0.14	<0.08
CTR on recommended experiences	7.8%	8.4%	8.0%
Completion rate after click	34%	22%	30%
Subscription conversion from recommenders	3.1%	1.9%	2.8%

The Problem

The recommendation engine appears optimized for clicks, but many clicked recommendations do not lead to meaningful family history engagement. Leadership wants to know whether the model is poorly calibrated, using the wrong objective, or underperforming for key user segments such as new users and older family-tree builders.

Requirements

Interpret the metrics and explain what they suggest about recommendation quality.
Diagnose the most likely causes of the gap between click performance and downstream outcomes.
Identify which additional slices or validation methods you would use.
Recommend specific model, threshold, or ranking changes.
Explain how you would measure success after changes are deployed.

Constraints

Home feed has only 5 recommendation slots.
False-positive recommendations reduce trust and weekly retention.
New users have sparse history for the first 7 days.
Product team can retrain the model only once every 2 weeks.

Context

Current Performance

Metric	Offline Validation	Online Last 30 Days	Target
Precision@5	0.41	0.38	0.45
Recall@5	0.29	0.24	0.30
F1@5	0.34	0.29	0.36
AUC-ROC (click prediction)	0.81	0.78	0.82
Log Loss	0.49	0.57	<0.50
Calibration error	0.06	0.14	<0.08
CTR on recommended experiences	7.8%	8.4%	8.0%
Completion rate after click	34%	22%	30%
Subscription conversion from recommenders	3.1%	1.9%	2.8%

The Problem

Requirements

Interpret the metrics and explain what they suggest about recommendation quality.
Diagnose the most likely causes of the gap between click performance and downstream outcomes.
Identify which additional slices or validation methods you would use.
Recommend specific model, threshold, or ranking changes.
Explain how you would measure success after changes are deployed.

Constraints

Home feed has only 5 recommendation slots.
False-positive recommendations reduce trust and weekly retention.
New users have sparse history for the first 7 days.
Product team can retrain the model only once every 2 weeks.

Context

Current Performance

Metric	Offline Validation	Online Last 30 Days	Target
Precision@5	0.41	0.38	0.45
Recall@5	0.29	0.24	0.30
F1@5	0.34	0.29	0.36
AUC-ROC (click prediction)	0.81	0.78	0.82
Log Loss	0.49	0.57	<0.50
Calibration error	0.06	0.14	<0.08
CTR on recommended experiences	7.8%	8.4%	8.0%
Completion rate after click	34%	22%	30%
Subscription conversion from recommenders	3.1%	1.9%	2.8%

The Problem

Requirements

Interpret the metrics and explain what they suggest about recommendation quality.
Diagnose the most likely causes of the gap between click performance and downstream outcomes.
Identify which additional slices or validation methods you would use.
Recommend specific model, threshold, or ranking changes.
Explain how you would measure success after changes are deployed.

Constraints

Home feed has only 5 recommendation slots.
False-positive recommendations reduce trust and weekly retention.
New users have sparse history for the first 7 days.
Product team can retrain the model only once every 2 weeks.

Context

Current Performance

Metric	Offline Validation	Online Last 30 Days	Target
Precision@5	0.41	0.38	0.45
Recall@5	0.29	0.24	0.30
F1@5	0.34	0.29	0.36
AUC-ROC (click prediction)	0.81	0.78	0.82
Log Loss	0.49	0.57	<0.50
Calibration error	0.06	0.14	<0.08
CTR on recommended experiences	7.8%	8.4%	8.0%
Completion rate after click	34%	22%	30%
Subscription conversion from recommenders	3.1%	1.9%	2.8%

The Problem

Requirements

Interpret the metrics and explain what they suggest about recommendation quality.
Diagnose the most likely causes of the gap between click performance and downstream outcomes.
Identify which additional slices or validation methods you would use.
Recommend specific model, threshold, or ranking changes.
Explain how you would measure success after changes are deployed.

Constraints

Home feed has only 5 recommendation slots.
False-positive recommendations reduce trust and weekly retention.
New users have sparse history for the first 7 days.
Product team can retrain the model only once every 2 weeks.

Interview Guides

Context

Current Performance

The Problem

Requirements

Constraints

Evaluate Family History Recommendations

Context

Current Performance

The Problem

Requirements

Constraints

Your Answer

Evaluate Family History Recommendations

Context

Current Performance

The Problem

Requirements

Constraints

Evaluate Family History Recommendations

Context

Current Performance

The Problem

Requirements

Constraints

Your Answer