Context
Asana is testing a redesign of the My Tasks surface that adds richer task previews and suggested next actions. Early readouts suggest users engage more with tasks, but there is concern the added UI may slow down actual task completion.
Hypothesis Seed
The product team believes the new My Tasks experience will increase meaningful engagement by helping users review and prioritize work, while preserving execution efficiency. You need to design and analyze an experiment where the treatment may raise engagement but reduce task completion speed.
Constraints
- Eligible traffic: 180,000 weekly active users per day on web and desktop who visit My Tasks
- Maximum experiment duration: 21 days; leadership wants a ship / don’t-ship recommendation by then
- Allocation can be ramped, but the steady-state target is 50/50
- Baseline task completion rate within 24 hours of task view: 32%
- Baseline median time from task view to completion: 18 hours; mean is 26 hours, standard deviation 40 hours due to heavy skew
- Business trade-off: a false positive is costly because slower completion can harm team productivity; a false negative is acceptable if it avoids shipping a harmful workflow change
Deliverables
- Define a clear null and alternative hypothesis, including whether the primary test should be one-sided or two-sided.
- Choose the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Be explicit about whether task completion speed is a guardrail or co-primary metric, and justify that choice.
- Calculate the required sample size and expected duration for an experiment powered to detect a 2% relative lift in the primary metric, while also monitoring a maximum tolerable 3% relative degradation in task completion speed.
- Specify the experiment design: unit of randomization, allocation, stratification, and pre-registered analysis plan, including how you will handle peeking, multiple comparisons, and any unit-of-analysis mismatch.
- Explain how you would interpret the result if the experiment shows a statistically significant increase in engagement but a statistically significant decrease in task completion speed, and state your final ship / don’t-ship rule.
