Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Onboarding Test With Upgrade Cannibalization

HardStatistics & Probability00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Business Context

Asana wants to test a new onboarding flow for newly created workspaces. The team expects better activation, but worries the flow could reduce immediate self-serve upgrades to Starter or Advanced by adding friction or delaying exposure to upgrade prompts.

Problem Statement

Design and analyze an experiment to determine whether the new onboarding flow improves activation without materially harming self-serve monetization.

Given Data

A 21-day randomized experiment was run at the new workspace level.

MetricControl: current onboardingTreatment: new onboarding
New workspaces52,40052,100
Activated within 7 days18,86420,840
Self-serve upgrades within 14 days3,1442,918
Mean 14-day revenue per workspace$9.60$9.18

Additional design inputs:

ParameterValue
Significance level for activation test0.05
One-sided significance level for cannibalization guardrail0.05
Non-inferiority margin for upgrade rate-0.003
Baseline activation rate0.360
Baseline upgrade rate0.060
Desired power0.80
Minimum detectable activation lift0.015

Requirements

  1. Define the experiment unit, primary metric, and guardrail metric.
  2. State hypotheses for:
    • activation improvement,
    • upgrade-rate non-inferiority.
  3. Run a two-proportion z-test for activation.
  4. Run a one-sided non-inferiority test for self-serve upgrade rate using margin Δ=−0.003\Delta = -0.003Δ=−0.003.
  5. Compute a 95% confidence interval for the activation lift and a 95% one-sided lower bound for the upgrade-rate difference.
  6. Based on the results, decide whether Asana should roll out, hold, or iterate.
  7. Briefly comment on whether the planned sample size was adequate for the activation metric.

Assumptions

  • Randomization is valid at workspace creation and each workspace appears once.
  • No major interference across workspaces.
  • Upgrade and activation outcomes are approximately independent across randomized units.
  • Use normal approximations for proportion tests.