Interview Guides

Tiny Revenue Lift Significance Test | Dataford Interview Questions - Dataford - Ace your Interview

Tiny Revenue Lift Significance Test

Medium

Statistics & Probability

Asked at 2 companies2A/B TestingConfidence IntervalsStatistical Significance

Also asked at

Problem

Business Context

StreamCart, a subscription video platform, tested a new upsell banner on its annual-plan checkout page. The experiment reached a very large audience, and the PM wants to know how to explain a result that is statistically significant but may not matter much financially.

Problem Statement

Use the experiment data below to determine whether the treatment increased average revenue per visitor (ARPV), and whether the effect is large enough to matter for the business.

Given Data

Group	Sample Size	Mean ARPV	Standard Deviation
Control	500000	12.40	40.00
Treatment	500000	12.48	40.00

Additional business inputs:

Metric	Value
Significance level	0.05
Minimum business-relevant lift per visitor	0.25
Monthly visitors to this page	8000000
One-time engineering + design rollout cost	180000

Requirements

State the null and alternative hypotheses for the difference in mean ARPV.
Compute the standard error for the difference in means.
Calculate the z-statistic and p-value.
Construct a 95% confidence interval for the treatment effect.
Compare statistical significance with practical significance using the business threshold of $0.25 per visitor.
Estimate the expected monthly incremental revenue if the observed lift holds.
Give a recommendation on whether StreamCart should roll out the banner broadly.

Assumptions

Random assignment was valid and independent across users.
ARPV is measured over the same attribution window in both groups.
With this sample size, the sampling distribution of the mean difference is approximately normal by the Central Limit Theorem.
Ignore downstream retention effects unless stated otherwise.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"control_sd":40,"treatment_n":500000,"rollout_cost":180000,"treatment_sd":40,"monthly_visitors":8000000,"control_mean_arpv":12.4,"treatment_mean_arpv":12.48,"minimum_business_lift":0.25}

Output(none)

Problem

Business Context

Problem Statement

Use the experiment data below to determine whether the treatment increased average revenue per visitor (ARPV), and whether the effect is large enough to matter for the business.

Given Data

Group	Sample Size	Mean ARPV	Standard Deviation
Control	500000	12.40	40.00
Treatment	500000	12.48	40.00

Additional business inputs:

Metric	Value
Significance level	0.05
Minimum business-relevant lift per visitor	0.25
Monthly visitors to this page	8000000
One-time engineering + design rollout cost	180000

Requirements

State the null and alternative hypotheses for the difference in mean ARPV.
Compute the standard error for the difference in means.
Calculate the z-statistic and p-value.
Construct a 95% confidence interval for the treatment effect.
Compare statistical significance with practical significance using the business threshold of $0.25 per visitor.
Estimate the expected monthly incremental revenue if the observed lift holds.
Give a recommendation on whether StreamCart should roll out the banner broadly.

Assumptions

Random assignment was valid and independent across users.
ARPV is measured over the same attribution window in both groups.
With this sample size, the sampling distribution of the mean difference is approximately normal by the Central Limit Theorem.
Ignore downstream retention effects unless stated otherwise.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"control_sd":40,"treatment_n":500000,"rollout_cost":180000,"treatment_sd":40,"monthly_visitors":8000000,"control_mean_arpv":12.4,"treatment_mean_arpv":12.48,"minimum_business_lift":0.25}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

AStatistical vs Business Impact in Pricing TestMedium

Experiment Significance for Pricing BannerMedium Significance Test for Signup LiftMedium

Next question

Tiny Revenue Lift Significance Test

Medium

Statistics & Probability

Asked at 2 companies2A/B TestingConfidence IntervalsStatistical Significance

Also asked at

Problem

Business Context

Problem Statement

Use the experiment data below to determine whether the treatment increased average revenue per visitor (ARPV), and whether the effect is large enough to matter for the business.

Given Data

Group	Sample Size	Mean ARPV	Standard Deviation
Control	500000	12.40	40.00
Treatment	500000	12.48	40.00

Additional business inputs:

Metric	Value
Significance level	0.05
Minimum business-relevant lift per visitor	0.25
Monthly visitors to this page	8000000
One-time engineering + design rollout cost	180000

Requirements

State the null and alternative hypotheses for the difference in mean ARPV.
Compute the standard error for the difference in means.
Calculate the z-statistic and p-value.
Construct a 95% confidence interval for the treatment effect.
Compare statistical significance with practical significance using the business threshold of $0.25 per visitor.
Estimate the expected monthly incremental revenue if the observed lift holds.
Give a recommendation on whether StreamCart should roll out the banner broadly.

Assumptions

Random assignment was valid and independent across users.
ARPV is measured over the same attribution window in both groups.
With this sample size, the sampling distribution of the mean difference is approximately normal by the Central Limit Theorem.
Ignore downstream retention effects unless stated otherwise.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"control_sd":40,"treatment_n":500000,"rollout_cost":180000,"treatment_sd":40,"monthly_visitors":8000000,"control_mean_arpv":12.4,"treatment_mean_arpv":12.48,"minimum_business_lift":0.25}

Output(none)

Problem

Business Context

Problem Statement

Use the experiment data below to determine whether the treatment increased average revenue per visitor (ARPV), and whether the effect is large enough to matter for the business.

Given Data

Group	Sample Size	Mean ARPV	Standard Deviation
Control	500000	12.40	40.00
Treatment	500000	12.48	40.00

Additional business inputs:

Metric	Value
Significance level	0.05
Minimum business-relevant lift per visitor	0.25
Monthly visitors to this page	8000000
One-time engineering + design rollout cost	180000

Requirements

State the null and alternative hypotheses for the difference in mean ARPV.
Compute the standard error for the difference in means.
Calculate the z-statistic and p-value.
Construct a 95% confidence interval for the treatment effect.
Compare statistical significance with practical significance using the business threshold of $0.25 per visitor.
Estimate the expected monthly incremental revenue if the observed lift holds.
Give a recommendation on whether StreamCart should roll out the banner broadly.

Assumptions

Random assignment was valid and independent across users.
ARPV is measured over the same attribution window in both groups.
With this sample size, the sampling distribution of the mean difference is approximately normal by the Central Limit Theorem.
Ignore downstream retention effects unless stated otherwise.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":500000,"control_sd":40,"treatment_n":500000,"rollout_cost":180000,"treatment_sd":40,"monthly_visitors":8000000,"control_mean_arpv":12.4,"treatment_mean_arpv":12.48,"minimum_business_lift":0.25}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

AStatistical vs Business Impact in Pricing TestMedium

Experiment Significance for Pricing BannerMedium Significance Test for Signup LiftMedium

Next question