Context
A UX designer proposes a redesigned Facebook profile header in the Facebook app to make key actions (Follow, Message, Add Friend) more prominent and reduce visual clutter. Before engineering fully builds and ships it, the team wants a rigorous online experiment to validate whether the design improves user behavior without harming downstream engagement.
Hypothesis Seed
The new profile header simplifies hierarchy and increases clarity of primary actions. The team believes this will increase the rate at which visitors take a meaningful action on another person's profile, especially on mobile, while not reducing session quality or causing accidental taps.
Constraints
- Eligible traffic: 1.2M profile-header impressions/day on Facebook mobile app
- Only 60% of impressions come from logged-in users with stable
user_id; the rest should be excluded from the primary analysis
- Maximum decision window: 14 days
- Engineering wants a go/no-go decision before the next app release
- False positives are costly because rebuilding the profile surface is expensive; false negatives are acceptable if the design can be iterated later
- The experiment must tolerate weekly seasonality and pre-register a plan that avoids peeking
Deliverables
- Define the experiment hypothesis, the primary metric, 2-4 guardrail metrics, and at least one secondary metric. State the MDE explicitly.
- Calculate the required sample size per arm using the supplied baseline assumptions, and convert it into an estimated runtime given available traffic.
- Choose the unit of randomization and explain why it is preferable to alternatives such as session-level randomization for this Facebook surface.
- Write a concise analysis plan: statistical test, alpha/power, peeking policy, multiple-comparison treatment, and how you will handle sample-ratio mismatch.
- State a clear ship / don't ship / iterate rule that respects both the primary metric and guardrails.
Use these baseline assumptions for the power analysis: current meaningful profile action rate = 12.0% per eligible profile viewer; target MDE = +3% relative; alpha = 0.05; power = 80%.
