Interview Guides

A/B Test Validation for Signup Redesign | Dataford Interview Questions - Dataford - Ace your Interview

A/B Test Validation for Signup Redesign

Easy

Statistics & Probability

Asked at 12 companies12A/B TestingStatistical SignificanceExperimentation

Also asked at

Problem

Business Context

StreamCart, a subscription video platform, changed its signup page to reduce friction and increase paid trial starts. The product team ran a randomized A/B test before deciding whether to launch the new design to all traffic.

Problem Statement

Use the experiment results to determine whether the new signup page improved conversion rate enough to justify rollout.

Given Data

Group	Users Exposed	Trial Starts	Conversion Rate
Control (old signup)	18,400	2,116	11.50%
Treatment (new signup)	18,900	2,325	12.30%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power for planning	0.80
Minimum detectable effect for planning	0.8 percentage points

Requirements

State the null and alternative hypotheses for the conversion-rate comparison.
Compute the observed conversion rates and the absolute lift.
Run a two-proportion z-test using the pooled standard error.
Calculate the two-sided p-value and determine whether the result is statistically significant at $\alpha = 0.05$ .
Construct a 95% confidence interval for the difference in conversion rates.
Assess whether the observed lift is practically meaningful for rollout.
Briefly explain how you would think about sample size adequacy for this test.

Assumptions

Users were randomly assigned and each user appears once.
Conversion is binary: a user either starts a paid trial or does not.
No major traffic-source mix shift occurred during the experiment.
The normal approximation is appropriate because both groups have large sample sizes and sufficient successes/failures.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18400,"treatment_n":18900,"control_rate":0.115,"power_target":0.8,"treatment_rate":0.123,"control_conversions":2116,"treatment_conversions":2325,"minimum_detectable_effect":0.008}

Output(none)

Problem

Business Context

Problem Statement

Use the experiment results to determine whether the new signup page improved conversion rate enough to justify rollout.

Given Data

Group	Users Exposed	Trial Starts	Conversion Rate
Control (old signup)	18,400	2,116	11.50%
Treatment (new signup)	18,900	2,325	12.30%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power for planning	0.80
Minimum detectable effect for planning	0.8 percentage points

Requirements

State the null and alternative hypotheses for the conversion-rate comparison.
Compute the observed conversion rates and the absolute lift.
Run a two-proportion z-test using the pooled standard error.
Calculate the two-sided p-value and determine whether the result is statistically significant at $\alpha = 0.05$ .
Construct a 95% confidence interval for the difference in conversion rates.
Assess whether the observed lift is practically meaningful for rollout.
Briefly explain how you would think about sample size adequacy for this test.

Assumptions

Users were randomly assigned and each user appears once.
Conversion is binary: a user either starts a paid trial or does not.
No major traffic-source mix shift occurred during the experiment.
The normal approximation is appropriate because both groups have large sample sizes and sufficient successes/failures.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18400,"treatment_n":18900,"control_rate":0.115,"power_target":0.8,"treatment_rate":0.123,"control_conversions":2116,"treatment_conversions":2325,"minimum_detectable_effect":0.008}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Conversion Significance for Signup RedesignEasy

A/B Test for Pricing PageEasy Conversion Significance for Signup RedesignMedium

Next question

A/B Test Validation for Signup Redesign

Easy

Statistics & Probability

Asked at 12 companies12A/B TestingStatistical SignificanceExperimentation

Also asked at

Problem

Business Context

Problem Statement

Use the experiment results to determine whether the new signup page improved conversion rate enough to justify rollout.

Given Data

Group	Users Exposed	Trial Starts	Conversion Rate
Control (old signup)	18,400	2,116	11.50%
Treatment (new signup)	18,900	2,325	12.30%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power for planning	0.80
Minimum detectable effect for planning	0.8 percentage points

Requirements

State the null and alternative hypotheses for the conversion-rate comparison.
Compute the observed conversion rates and the absolute lift.
Run a two-proportion z-test using the pooled standard error.
Calculate the two-sided p-value and determine whether the result is statistically significant at $\alpha = 0.05$ .
Construct a 95% confidence interval for the difference in conversion rates.
Assess whether the observed lift is practically meaningful for rollout.
Briefly explain how you would think about sample size adequacy for this test.

Assumptions

Users were randomly assigned and each user appears once.
Conversion is binary: a user either starts a paid trial or does not.
No major traffic-source mix shift occurred during the experiment.
The normal approximation is appropriate because both groups have large sample sizes and sufficient successes/failures.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18400,"treatment_n":18900,"control_rate":0.115,"power_target":0.8,"treatment_rate":0.123,"control_conversions":2116,"treatment_conversions":2325,"minimum_detectable_effect":0.008}

Output(none)

Problem

Business Context

Problem Statement

Use the experiment results to determine whether the new signup page improved conversion rate enough to justify rollout.

Given Data

Group	Users Exposed	Trial Starts	Conversion Rate
Control (old signup)	18,400	2,116	11.50%
Treatment (new signup)	18,900	2,325	12.30%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power for planning	0.80
Minimum detectable effect for planning	0.8 percentage points

Requirements

State the null and alternative hypotheses for the conversion-rate comparison.
Compute the observed conversion rates and the absolute lift.
Run a two-proportion z-test using the pooled standard error.
Calculate the two-sided p-value and determine whether the result is statistically significant at $\alpha = 0.05$ .
Construct a 95% confidence interval for the difference in conversion rates.
Assess whether the observed lift is practically meaningful for rollout.
Briefly explain how you would think about sample size adequacy for this test.

Assumptions

Users were randomly assigned and each user appears once.
Conversion is binary: a user either starts a paid trial or does not.
No major traffic-source mix shift occurred during the experiment.
The normal approximation is appropriate because both groups have large sample sizes and sufficient successes/failures.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18400,"treatment_n":18900,"control_rate":0.115,"power_target":0.8,"treatment_rate":0.123,"control_conversions":2116,"treatment_conversions":2325,"minimum_detectable_effect":0.008}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Conversion Significance for Signup RedesignEasy

A/B Test for Pricing PageEasy Conversion Significance for Signup RedesignMedium

Next question