Context
BrightCart, a DTC e-commerce company, wants to redesign its paid-traffic landing page. The growth team has created three new variants and wants to test them against the current page at the same time.
Hypothesis Seed
Variant B emphasizes social proof, Variant C simplifies the form, and Variant D highlights a limited-time offer. The team believes at least one variant will improve purchase conversion rate, but they are concerned about false positives from testing multiple variants simultaneously.
Constraints
- Eligible traffic: 120,000 unique landing-page visitors per day from paid channels
- Maximum experiment window: 14 days, after which the ad campaign creative must be finalized
- Baseline purchase conversion rate: 8.0%
- Business-relevant lift: at least 8% relative improvement in conversion rate
- False positives are costly because rolling out a weak page would increase media spend inefficiency; false negatives are also meaningful because paid traffic is expensive and the team does not want to miss a real winner
- Engineering can support user-level randomization and persistent bucketing across visits
Deliverables
- Define the null and alternative hypotheses for a 4-arm experiment (Control, B, C, D), including how you will handle multiple comparisons.
- Specify the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Include the MDE explicitly.
- Calculate the required sample size per arm using real numbers, then translate that into expected runtime given the available traffic.
- Choose the unit of randomization, allocation, and duration, and explain why that design is appropriate.
- Pre-register an analysis plan covering the statistical test, correction for multiple comparisons, peeking policy, and a clear ship / don’t-ship rule that respects guardrails.
