Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Diagnose Misleading Signup A/B Test

MediumStatistics & Probability00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Business Context

StreamCart, a subscription video platform, tested a new homepage signup CTA. After 10 days, the product manager wants to ship because treatment looks better on the primary metric.

Problem Statement

You need to determine whether the observed lift in signup conversion is statistically significant and whether the experiment itself may be misleading due to common A/B testing pitfalls.

Given Data

MetricControlTreatment
Assigned users52,40047,600
Signups6,0265,950
Observed conversion rate11.50%12.50%
Expected traffic split50%50%

Additional experiment facts:

ItemValue
Test duration10 days
Significance level0.05
Number of dashboard checks during run8
Other metrics reviewed post hoc6
Day 1 treatment conversion rate14.2%
Days 2-10 treatment conversion rate12.3%

Requirements

  1. State the null and alternative hypotheses for signup conversion.
  2. Run a two-proportion z-test on the primary metric.
  3. Compute a 95% confidence interval for the treatment-control difference.
  4. Check for sample ratio mismatch (SRM) using the expected 50/50 split.
  5. Based on the numbers above, identify the main pitfalls that could make this A/B test misleading.
  6. Give a recommendation on whether StreamCart should roll out immediately.

Assumptions

  • User-level randomization was intended.
  • Each assigned user is counted once.
  • Signup is a binary outcome.
  • Use a two-sided test for the primary metric.
  • For the SRM check, use a chi-square goodness-of-fit test with expected counts based on a 50/50 split.