Explain P-Value and Power in A/B Testing | Dataford Interview Questions

Your question is Explain P-Value and Power in A/B Testing. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

PixelCraft, a design collaboration app, tested a new onboarding screen created by the Product Design team. The designer wants a plain-language explanation of p-value and statistical power, grounded in the actual experiment results.

Problem Statement

You need to explain whether the new onboarding improved activation, what the p-value says about the observed result, and what statistical power says about the test's ability to detect a meaningful improvement.

Given Data

Metric	Control	Treatment
Users shown variant	8,400	8,600
Activated within 7 days	1,260	1,376
Activation rate	15.0%	16.0%

Additional assumptions for power analysis:

Parameter	Value
Significance level	0.05
Test type	Two-sided two-proportion z-test
Minimum meaningful lift	1.5 percentage points
Baseline activation rate	15.0%

Requirements

State the null and alternative hypotheses.
Calculate the observed difference in activation rates.
Compute the test statistic and p-value.
Explain the p-value in plain English for a non-technical Product Designer.
Estimate the statistical power to detect a lift from 15.0% to 16.5% with the given sample sizes.
Explain power in plain English and say whether this experiment was adequately powered.
Give a business recommendation on whether to ship, rerun, or collect more data.

Assumptions

Users were randomly assigned to control and treatment.
Each user appears once in the analysis.
Normal approximation is appropriate because sample sizes are large.
Ignore segmentation and multiple-testing issues for this exercise.