Onboarding Test With Upgrade Cannibalization

Business Context

Asana wants to test a new onboarding flow for newly created workspaces. The team expects better activation, but worries the flow could reduce immediate self-serve upgrades to Starter or Advanced by adding friction or delaying exposure to upgrade prompts.

Problem Statement

Design and analyze an experiment to determine whether the new onboarding flow improves activation without materially harming self-serve monetization.

Given Data

A 21-day randomized experiment was run at the new workspace level.

Additional design inputs:

Requirements

1. Define the experiment unit, primary metric, and guardrail metric.
2. State hypotheses for:
   • activation improvement,
   • upgrade-rate non-inferiority.
3. Run a two-proportion z-test for activation.
4. Run a one-sided non-inferiority test for self-serve upgrade rate using margin $\Delta = -0.003$.
5. Compute a 95% confidence interval for the activation lift and a 95% one-sided lower bound for the upgrade-rate difference.
6. Based on the results, decide whether Asana should roll out, hold, or iterate.
7. Briefly comment on whether the planned sample size was adequate for the activation metric.

Assumptions

• Randomization is valid at workspace creation and each workspace appears once.
• No major interference across workspaces.
• Upgrade and activation outcomes are approximately independent across randomized units.
• Use normal approximations for proportion tests.