Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Primary vs Guardrail Metric Choice

Medium
A/B Testing & ExperimentationExperimentationGuardrail MetricsA/B Testing
Asked 1mo ago|
Roku
Roku
Asked 44 times

Problem

Scenario

You are planning an A/B test for a mobile game feature and need to decide how success will be measured. The feature may improve short-term engagement, but there is concern it could also hurt longer-term player experience or monetization in ways that would be missed if you optimize for only one outcome.

Question

How do you decide which metric should be the primary metric and which metrics should be guardrails in this experiment? Walk through how you would make that choice, how it affects experiment design, and how you would interpret the results if the primary metric improves but a guardrail worsens.

Metric Setup

MDE·2% relative liftUnit·player_idGuardrails·ARPDAU, crash-free session rate, D7 retentionPrimary metric·D1 session-start rate

You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.

Sign up freeI have an account
Sign up to unlock solutions
Niantic Data Scientist Interview QuestionsRedesign Health Data Scientist Interview QuestionsWix Product Manager Interview QuestionsZynga Data Scientist Interview QuestionsClouddevs Mobile Engineer Interview Questions
Next questions
Sun LifePrimary vs Guardrail MetricsEasyVolvo CarsDefine Metrics for a Customer TestHardEchostarDiscovery UI Metrics and GuardrailsMedium
0 / ~200 words