Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Explaining P-Value and Power

MediumStatistics & Probability00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Business Context

PixelCraft, a design collaboration app, tested a new onboarding illustration intended to increase first-session completion. A product designer saw that the experiment result was “not significant” and asked what the p-value means, and why the team also cares about statistical power.

Problem Statement

Use the experiment data below to explain both p-value and statistical power in plain language, while also computing the formal test result. Then assess whether the test was large enough to reliably detect the target improvement.

Given Data

MetricValue
Control users4,800
Control completions1,200
Control completion rate25.0%
Treatment users4,900
Treatment completions1,274
Treatment completion rate26.0%
Observed lift1.0 percentage point
Significance level0.05
Desired power80%
Minimum detectable effect (absolute)2.0 percentage points

Assume a two-sided test comparing two independent proportions.

Requirements

  1. State the null and alternative hypotheses.
  2. Compute the pooled proportion, standard error, z-statistic, and p-value.
  3. Decide whether the observed lift is statistically significant at α=0.05\alpha = 0.05α=0.05.
  4. Explain the p-value in plain English for a non-technical product designer.
  5. Estimate whether the current test had enough power to detect a 2.0 percentage point lift.
  6. Explain statistical power in plain English and why a non-significant result does not necessarily mean “no effect.”

Assumptions

  • Users were randomly assigned.
  • Each user is counted once.
  • The normal approximation for proportions is appropriate.
  • Use equal-variance planning assumptions for the power/sample size check.