Business Context

FitMeal, a subscription meal-planning app, tested a new onboarding flow intended to increase paid trial starts. After a 21-day A/B test, the treatment looked better than control, but the analytics dashboard flagged the result as not statistically significant.

Problem Statement

Evaluate whether the experiment provides enough evidence to ship the new onboarding flow, and explain how you would assess a directionally positive but statistically non-significant result.

Given Data

Additional inputs:

Requirements

1. State the null and alternative hypotheses.
2. Run a two-proportion z-test using the observed data.
3. Compute a 95% confidence interval for the treatment-control difference.
4. Determine whether the result is statistically significant at the 5% level.
5. Compare the observed lift with the planned MDE and explain what that implies about power.
6. Recommend what FitMeal should do next: ship, stop, or continue testing, with justification.

Assumptions

• Users were randomly assigned at the user level.
• Each user appears once and outcomes are independent across users.
• No major sample ratio mismatch or instrumentation issues were detected.
• The primary metric is paid trial start rate, and no interim peeking adjustment is required.