Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Measure Onboarding Message Impact

MediumA/B Testing & Experimentation00:00
Airtel Payments Bank
Your interviewer · Data Scientist
In session
Interviewer

Welcome to your interview for the Data Scientist role at Airtel Payments Bank.

The question is on your right: Measure Onboarding Message Impact. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen. You have three graded attempts to score 7/10 or better.

Only Submit answer is graded - discussion is free practice.

Problem

Scenario

You work on a consumer product where new users see an onboarding message during their first experience. The team believes the message will help users understand the product faster and become more engaged, but it is not clear whether any observed change would reflect real behavior change or just noise.

Question

How would you measure whether the new onboarding message actually changed user behavior? Walk through the experiment you would run, what success metric you would use, and how you would decide whether to launch it.

What Strong Answers Cover

  • A causal experiment design rather than before/after comparison
  • A primary metric tied to meaningful post-onboarding behavior
  • Guardrails that protect signup quality and retention
  • Power analysis with an explicit minimum detectable effect