Scenario

You work on a consumer fintech app and are testing a redesigned card-activation prompt shown to newly approved members in the mobile app. The product manager believes the new prompt will increase activation completion by making the next step clearer, and after two days says the treatment is already statistically significant and wants to stop the test early. You need to decide whether that recommendation is valid and how the experiment should be run and analyzed.

Constraints

• Eligible traffic: 120,000 newly approved members per day
• Baseline activation completion rate within 7 days: 40%
• Maximum experiment duration: 14 days
• 50/50 allocation unless you justify otherwise
• Customer support contact rate must not increase by more than 0.3 percentage points

Question

How would you design and analyze this experiment, and what would you recommend when asked to stop after two days because the result already looks significant? Explain the decision rule you would pre-register, including how you would handle early stopping, guardrails, and any validity checks before making a ship recommendation.