You work on a work-management product and ran an A/B test on a growth change for two weeks. The result is not statistically significant, but the PM argues the directional lift is enough and wants to launch anyway. You need to advise whether the evidence supports shipping.
How would you respond to the PM, and what framework would you use to decide whether to ship, extend, or stop the experiment? Explain how statistical power, minimum detectable effect, and guardrail metrics affect your recommendation.
Interpreting non-significant A/B test resultsUsing power and MDE to distinguish inconclusive from negative resultsApplying guardrails to launch decisionsAvoiding peeking and post-hoc decision-making