Interview Guides

Conversion Lift Test for Signup Feature | Dataford Interview Questions - Dataford - Ace your Interview

Conversion Lift Test for Signup Feature

Medium

Statistics & Probability

Asked at 4 companies4Hypothesis TestingA/B TestingStatistical Significance

Also asked at

Problem

Business Context

StreamCart, a subscription video platform, launched a new signup page feature that shows personalized plan recommendations. The product team wants to know whether the feature improves paid conversion from visitor to subscriber.

Problem Statement

You are asked to design and analyze an experiment to determine whether the new feature increases conversion rate relative to the current signup flow.

Given Data

A 14-day user-level A/B test was run with random assignment at the first eligible visit.

Group	Users Exposed	Paid Conversions	Conversion Rate
Control (current signup flow)	52,400	6,131	11.70%
Treatment (personalized recommendations)	51,900	6,492	12.51%

Additional parameters:

Parameter	Value
Significance level	0.05
Test type	One-tailed
Baseline conversion rate for planning	11.7%
Minimum detectable effect	0.8 percentage points
Desired power	80%

Requirements

State the null and alternative hypotheses for this experiment.
Identify the correct unit of randomization and primary metric.
Compute the observed lift in conversion rate.
Run a two-proportion z-test and calculate the p-value.
Construct a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at $\alpha = 0.05$ .
Check whether the observed sample size is roughly adequate for the stated minimum detectable effect.
Briefly list two guardrail metrics or experimental risks you would monitor before rollout.

Assumptions

Randomization was implemented correctly and independently across users.
Each user appears once in the analysis dataset.
No major tracking bugs or sample ratio mismatch occurred.
The experiment duration is long enough to cover day-of-week effects.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":51900,"control_rate":0.117,"mde_absolute":0.008,"baseline_rate":0.117,"treatment_rate":0.1251,"test_duration_days":14,"control_conversions":6131,"treatment_conversions":6492}

Output(none)

Problem

Business Context

Problem Statement

You are asked to design and analyze an experiment to determine whether the new feature increases conversion rate relative to the current signup flow.

Given Data

A 14-day user-level A/B test was run with random assignment at the first eligible visit.

Group	Users Exposed	Paid Conversions	Conversion Rate
Control (current signup flow)	52,400	6,131	11.70%
Treatment (personalized recommendations)	51,900	6,492	12.51%

Additional parameters:

Parameter	Value
Significance level	0.05
Test type	One-tailed
Baseline conversion rate for planning	11.7%
Minimum detectable effect	0.8 percentage points
Desired power	80%

Requirements

State the null and alternative hypotheses for this experiment.
Identify the correct unit of randomization and primary metric.
Compute the observed lift in conversion rate.
Run a two-proportion z-test and calculate the p-value.
Construct a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at $\alpha = 0.05$ .
Check whether the observed sample size is roughly adequate for the stated minimum detectable effect.
Briefly list two guardrail metrics or experimental risks you would monitor before rollout.

Assumptions

Randomization was implemented correctly and independently across users.
Each user appears once in the analysis dataset.
No major tracking bugs or sample ratio mismatch occurred.
The experiment duration is long enough to cover day-of-week effects.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":51900,"control_rate":0.117,"mde_absolute":0.008,"baseline_rate":0.117,"treatment_rate":0.1251,"test_duration_days":14,"control_conversions":6131,"treatment_conversions":6492}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Conversion Lift Significance for Signup RedesignMedium Conversion Significance for Signup RedesignMedium

A/B Test Validation for Signup RedesignEasy

Next question

Conversion Lift Test for Signup Feature

Medium

Statistics & Probability

Asked at 4 companies4Hypothesis TestingA/B TestingStatistical Significance

Also asked at

Problem

Business Context

Problem Statement

You are asked to design and analyze an experiment to determine whether the new feature increases conversion rate relative to the current signup flow.

Given Data

A 14-day user-level A/B test was run with random assignment at the first eligible visit.

Group	Users Exposed	Paid Conversions	Conversion Rate
Control (current signup flow)	52,400	6,131	11.70%
Treatment (personalized recommendations)	51,900	6,492	12.51%

Additional parameters:

Parameter	Value
Significance level	0.05
Test type	One-tailed
Baseline conversion rate for planning	11.7%
Minimum detectable effect	0.8 percentage points
Desired power	80%

Requirements

State the null and alternative hypotheses for this experiment.
Identify the correct unit of randomization and primary metric.
Compute the observed lift in conversion rate.
Run a two-proportion z-test and calculate the p-value.
Construct a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at $\alpha = 0.05$ .
Check whether the observed sample size is roughly adequate for the stated minimum detectable effect.
Briefly list two guardrail metrics or experimental risks you would monitor before rollout.

Assumptions

Randomization was implemented correctly and independently across users.
Each user appears once in the analysis dataset.
No major tracking bugs or sample ratio mismatch occurred.
The experiment duration is long enough to cover day-of-week effects.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":51900,"control_rate":0.117,"mde_absolute":0.008,"baseline_rate":0.117,"treatment_rate":0.1251,"test_duration_days":14,"control_conversions":6131,"treatment_conversions":6492}

Output(none)

Problem

Business Context

Problem Statement

You are asked to design and analyze an experiment to determine whether the new feature increases conversion rate relative to the current signup flow.

Given Data

A 14-day user-level A/B test was run with random assignment at the first eligible visit.

Group	Users Exposed	Paid Conversions	Conversion Rate
Control (current signup flow)	52,400	6,131	11.70%
Treatment (personalized recommendations)	51,900	6,492	12.51%

Additional parameters:

Parameter	Value
Significance level	0.05
Test type	One-tailed
Baseline conversion rate for planning	11.7%
Minimum detectable effect	0.8 percentage points
Desired power	80%

Requirements

State the null and alternative hypotheses for this experiment.
Identify the correct unit of randomization and primary metric.
Compute the observed lift in conversion rate.
Run a two-proportion z-test and calculate the p-value.
Construct a 95% confidence interval for the treatment-control difference.
Determine whether the result is statistically significant at $\alpha = 0.05$ .
Check whether the observed sample size is roughly adequate for the stated minimum detectable effect.
Briefly list two guardrail metrics or experimental risks you would monitor before rollout.

Assumptions

Randomization was implemented correctly and independently across users.
Each user appears once in the analysis dataset.
No major tracking bugs or sample ratio mismatch occurred.
The experiment duration is long enough to cover day-of-week effects.

Sample Data

Example 1

Input

{"alpha":0.05,"power":0.8,"control_n":52400,"treatment_n":51900,"control_rate":0.117,"mde_absolute":0.008,"baseline_rate":0.117,"treatment_rate":0.1251,"test_duration_days":14,"control_conversions":6131,"treatment_conversions":6492}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Conversion Lift Significance for Signup RedesignMedium Conversion Significance for Signup RedesignMedium

A/B Test Validation for Signup RedesignEasy

Next question