Retention A/B Test for Onboarding

Business Context

PulseFit, a mobile fitness app, launched a new personalized onboarding flow intended to help new users find relevant workouts faster. The product team wants to know whether the feature improves 7-day retention for newly registered users.

Problem Statement

You are asked to structure the experiment and evaluate whether the new feature increased retention relative to the current onboarding experience.

Given Data

A 14-day randomized A/B test was run on new users only, with user-level randomization at signup.

Additional planning inputs:

Requirements

1. Define the experimental design: unit of randomization, target population, primary metric, and at least two guardrail metrics.
2. State the null and alternative hypotheses for testing retention lift.
3. Compute the observed retention rates, pooled proportion, standard error, z-statistic, and two-sided p-value.
4. Construct a 95% confidence interval for the difference in retention rates.
5. Compare the observed effect to the minimum detectable effect and state whether the result is practically meaningful.
6. Conclude whether PulseFit should roll out the feature now, continue testing, or reject the feature.

Assumptions

• Randomization was implemented correctly with no sample ratio mismatch.
• Each user appears once and outcomes are independent across users.
• Day-7 retention is defined as at least one session on day 7 after signup.
• No major concurrent launches affected only one variant during the test window.