Interview Guides

Statistical vs Business Impact in Pricing Test | Dataford Interview Questions - Dataford - Ace your Interview

Statistical vs Business Impact in Pricing Test

Medium

Statistics & Probability

Asked at 2 companies2A/B TestingConfidence IntervalsStatistical Significance

Also asked at

Problem

Business Context

StreamCart, a grocery delivery app, tested a new pricing page that highlights annual-plan savings. The experiment ran on a large share of traffic, and the analytics dashboard shows a statistically significant lift in annual-plan conversion.

Problem Statement

You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.

Given Data

Group	Sample Size	Annual Plan Conversions	Conversion Rate
Control (old pricing page)	500,000	20,000	4.00%
Treatment (new pricing page)	500,000	20,750	4.15%

Additional business inputs:

Metric	Value
Absolute lift in conversion rate	0.15 percentage points
Relative lift	3.75%
Incremental gross profit per annual-plan conversion	$18
One-time engineering + design rollout cost	$220,000
Significance level	0.05

Requirements

State the null and alternative hypotheses for a two-sided test.
Compute the pooled conversion rate and the standard error.
Calculate the z-statistic and p-value for the difference in proportions.
Construct a 95% confidence interval for the absolute lift.
Estimate the expected incremental gross profit from the observed lift over these 1,000,000 users.
Explain how the result can be statistically significant but still have little business impact.
Recommend whether StreamCart should roll out the change based on both statistical and practical significance.

Assumptions

Random assignment was valid and there was no sample ratio mismatch.
Each user had one exposure and one conversion opportunity.
Gross profit per additional annual-plan conversion is constant at $18.
Ignore longer-term retention effects unless explicitly discussed.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"treatment_n":500000,"control_rate":0.04,"rollout_cost":220000,"absolute_lift":0.0015,"relative_lift":0.0375,"treatment_rate":0.0415,"control_conversions":20000,"treatment_conversions":20750,"incremental_gross_profit_per_conversion":18}

Output(none)

Problem

Business Context

Problem Statement

You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.

Given Data

Group	Sample Size	Annual Plan Conversions	Conversion Rate
Control (old pricing page)	500,000	20,000	4.00%
Treatment (new pricing page)	500,000	20,750	4.15%

Additional business inputs:

Metric	Value
Absolute lift in conversion rate	0.15 percentage points
Relative lift	3.75%
Incremental gross profit per annual-plan conversion	$18
One-time engineering + design rollout cost	$220,000
Significance level	0.05

Requirements

State the null and alternative hypotheses for a two-sided test.
Compute the pooled conversion rate and the standard error.
Calculate the z-statistic and p-value for the difference in proportions.
Construct a 95% confidence interval for the absolute lift.
Estimate the expected incremental gross profit from the observed lift over these 1,000,000 users.
Explain how the result can be statistically significant but still have little business impact.
Recommend whether StreamCart should roll out the change based on both statistical and practical significance.

Assumptions

Random assignment was valid and there was no sample ratio mismatch.
Each user had one exposure and one conversion opportunity.
Gross profit per additional annual-plan conversion is constant at $18.
Ignore longer-term retention effects unless explicitly discussed.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"treatment_n":500000,"control_rate":0.04,"rollout_cost":220000,"absolute_lift":0.0015,"relative_lift":0.0375,"treatment_rate":0.0415,"control_conversions":20000,"treatment_conversions":20750,"incremental_gross_profit_per_conversion":18}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Experiment Significance for Pricing BannerMedium

Tiny Revenue Lift Significance TestMedium

A/B Test for Pricing PageEasy

Next question

Statistical vs Business Impact in Pricing Test

Medium

Statistics & Probability

Asked at 2 companies2A/B TestingConfidence IntervalsStatistical Significance

Also asked at

Problem

Business Context

Problem Statement

You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.

Given Data

Group	Sample Size	Annual Plan Conversions	Conversion Rate
Control (old pricing page)	500,000	20,000	4.00%
Treatment (new pricing page)	500,000	20,750	4.15%

Additional business inputs:

Metric	Value
Absolute lift in conversion rate	0.15 percentage points
Relative lift	3.75%
Incremental gross profit per annual-plan conversion	$18
One-time engineering + design rollout cost	$220,000
Significance level	0.05

Requirements

State the null and alternative hypotheses for a two-sided test.
Compute the pooled conversion rate and the standard error.
Calculate the z-statistic and p-value for the difference in proportions.
Construct a 95% confidence interval for the absolute lift.
Estimate the expected incremental gross profit from the observed lift over these 1,000,000 users.
Explain how the result can be statistically significant but still have little business impact.
Recommend whether StreamCart should roll out the change based on both statistical and practical significance.

Assumptions

Random assignment was valid and there was no sample ratio mismatch.
Each user had one exposure and one conversion opportunity.
Gross profit per additional annual-plan conversion is constant at $18.
Ignore longer-term retention effects unless explicitly discussed.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"treatment_n":500000,"control_rate":0.04,"rollout_cost":220000,"absolute_lift":0.0015,"relative_lift":0.0375,"treatment_rate":0.0415,"control_conversions":20000,"treatment_conversions":20750,"incremental_gross_profit_per_conversion":18}

Output(none)

Problem

Business Context

Problem Statement

You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.

Given Data

Group	Sample Size	Annual Plan Conversions	Conversion Rate
Control (old pricing page)	500,000	20,000	4.00%
Treatment (new pricing page)	500,000	20,750	4.15%

Additional business inputs:

Metric	Value
Absolute lift in conversion rate	0.15 percentage points
Relative lift	3.75%
Incremental gross profit per annual-plan conversion	$18
One-time engineering + design rollout cost	$220,000
Significance level	0.05

Requirements

State the null and alternative hypotheses for a two-sided test.
Compute the pooled conversion rate and the standard error.
Calculate the z-statistic and p-value for the difference in proportions.
Construct a 95% confidence interval for the absolute lift.
Estimate the expected incremental gross profit from the observed lift over these 1,000,000 users.
Explain how the result can be statistically significant but still have little business impact.
Recommend whether StreamCart should roll out the change based on both statistical and practical significance.

Assumptions

Random assignment was valid and there was no sample ratio mismatch.
Each user had one exposure and one conversion opportunity.
Gross profit per additional annual-plan conversion is constant at $18.
Ignore longer-term retention effects unless explicitly discussed.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"treatment_n":500000,"control_rate":0.04,"rollout_cost":220000,"absolute_lift":0.0015,"relative_lift":0.0375,"treatment_rate":0.0415,"control_conversions":20000,"treatment_conversions":20750,"incremental_gross_profit_per_conversion":18}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Experiment Significance for Pricing BannerMedium

Tiny Revenue Lift Significance TestMedium

A/B Test for Pricing PageEasy

Next question