Context
Aircall’s Growth team has two high-priority experiments targeting the same new-workspace onboarding flow in the Aircall web app. Only one can use most of the eligible traffic unless you redesign the test plan.
Hypothesis Seed
Experiment A changes the Aircall workspace setup checklist to emphasize faster phone-number activation. The PM believes this will increase the share of new workspaces that complete setup and place a first outbound call.
Experiment B changes the Aircall in-app call-routing setup wizard to reduce friction when assigning numbers and teammates. The PM believes this will improve activation quality and reduce early support contacts.
Both experiments affect the same user journey and likely the same outcome metrics, so running them independently on overlapping traffic may create interference and make attribution ambiguous.
Constraints
- Eligible traffic: 12,000 new workspaces per week entering onboarding
- Average onboarding completion rate: 38% within 14 days
- Average “first outbound call within 14 days” rate: 24%
- Maximum decision window: 4 weeks from launch
- False positive cost is high: shipping a bad onboarding change can hurt activation and increase support load
- False negative cost is moderate: delaying a good change by a few weeks is acceptable
- Engineering can support either: (a) one clean A/B test, or (b) a factorial design if justified
Task
- Propose how you would handle the traffic conflict: prioritize one experiment, split traffic, run a factorial, or sequence them. State the decision criteria.
- Define the primary metric, 2-4 guardrails, and an explicit MDE for the chosen design.
- Calculate the required sample size and show whether the design fits within the 4-week traffic budget.
- Choose the unit of randomization, allocation, duration, and analysis plan, including peeking and multiple-comparison policy.
- Call out the main experimentation risks specific to overlapping onboarding changes at Aircall and explain how you would mitigate them.
