Website Design A/B Testing

如何测试哪种网站设计好？ (How to test which website design is better?) Asked in the Phone Screen stage.

You are comparing two homepage designs for a Google product surface. The goal is to decide whether the new design should replace the current one without hurting user experience or downstream engagement.

Assume the team can send 200,000 eligible users per day to the experiment and must decide within 14 days. The new design may improve clicks, but it could also increase bounce rate or slow page load.

Please answer:

1. State the null and alternative hypotheses.
2. Define one primary metric and 2 to 3 guardrail metrics, including the unit of analysis and a realistic MDE.
3. Estimate sample size and test duration with explicit assumptions.
4. Choose the unit of randomization and explain why.
5. Give a pre-registered analysis plan, including how you would handle peeking, multiple comparisons, and sample ratio mismatch.
6. List the main pitfalls you would watch for, such as novelty effects, SUTVA violations, or network interference.