Context
Asana is considering a change to the new-user onboarding flow in the web app: replacing the current project-creation prompt with a guided setup that recommends a starter project template and nudges users to invite a teammate earlier. The Growth team needs to decide whether the experiment can reach a launch decision within a fixed time window.
Hypothesis Seed
The team believes the guided onboarding will increase activation by helping new workspaces reach value faster. However, an overly aggressive invite prompt could reduce completion of onboarding or create low-quality invites, so the launch decision must balance activation gains against user-experience guardrails.
Constraints
- Eligible traffic: 18,000 new Asana workspace creators per day on web
- Maximum experiment window: 21 days, including ramp
- Planned allocation: 50/50 after a 1-day 5% ramp for instrumentation checks
- Baseline 7-day activation rate: 32% of new workspace creators complete all of the following within 7 days: create a project, add at least 3 tasks, and invite at least 1 teammate
- The PM says a lift smaller than 2 percentage points absolute is not worth shipping because of engineering and lifecycle-maintenance cost
- False positives are costly because a bad launch affects first impressions and downstream retention; false negatives are also costly because onboarding is a major growth lever
Deliverables
- State the null and alternative hypotheses, and define the primary metric, secondary metrics, and guardrails for this Asana onboarding experiment.
- Calculate the minimum sample size per arm needed for a launch decision using the stated baseline, alpha, power, and MDE. Translate that into expected runtime given the traffic constraint.
- Choose the unit of randomization and explain whether your unit of analysis matches it. If not, explain how you would analyze correctly.
- Pre-register an analysis plan: statistical test, peeking policy, handling of multiple metrics, and what you would do if you observe sample ratio mismatch.
- Give a clear ship / don’t-ship / iterate rule that respects both the primary metric and guardrails.
