Interview Guides

A/B Test for Onboarding Feature | Dataford Interview Questions - Dataford - Ace your Interview

A/B Test for Onboarding Feature

Medium

Statistics & Probability

Hypothesis TestingA/B TestingStatistical Significance

Problem

Business Context

FinPilot, a personal finance app, launched a new in-app onboarding checklist intended to improve first-week activation. The product team ran a 14-day user-level A/B test and wants to know whether the feature should be rolled out broadly.

Problem Statement

Evaluate whether the new onboarding checklist increased activation rate relative to the current experience, and assess whether the observed lift is large enough to matter operationally.

Given Data

Group	Users	Activated Users	Activation Rate
Control	24,800	4,712	19.0%
Treatment	25,150	5,156	20.5%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power	0.80
Minimum detectable effect to evaluate	1.0 percentage point

Requirements

State the null and alternative hypotheses for the activation-rate comparison.
Compute the sample proportions, pooled proportion, and standard error for a two-proportion z-test.
Calculate the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the treatment-minus-control lift.
Decide whether the result is statistically significant at the 5% level.
Compare the observed lift with the 1.0 percentage point practical threshold and comment on rollout.
Briefly explain how you would think about power and whether this test appears adequately sized.

Assumptions

Random assignment was done at the user level.
Each user appears once in the experiment dataset.
No major instrumentation issues or sample-ratio mismatch were detected.
Activation is measured consistently within 7 days for both groups.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25150,"control_rate":0.19,"mde_absolute":0.01,"power_target":0.8,"treatment_rate":0.205,"control_activated":4712,"treatment_activated":5156}

Output(none)

Problem

Business Context

Problem Statement

Evaluate whether the new onboarding checklist increased activation rate relative to the current experience, and assess whether the observed lift is large enough to matter operationally.

Given Data

Group	Users	Activated Users	Activation Rate
Control	24,800	4,712	19.0%
Treatment	25,150	5,156	20.5%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power	0.80
Minimum detectable effect to evaluate	1.0 percentage point

Requirements

State the null and alternative hypotheses for the activation-rate comparison.
Compute the sample proportions, pooled proportion, and standard error for a two-proportion z-test.
Calculate the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the treatment-minus-control lift.
Decide whether the result is statistically significant at the 5% level.
Compare the observed lift with the 1.0 percentage point practical threshold and comment on rollout.
Briefly explain how you would think about power and whether this test appears adequately sized.

Assumptions

Random assignment was done at the user level.
Each user appears once in the experiment dataset.
No major instrumentation issues or sample-ratio mismatch were detected.
Activation is measured consistently within 7 days for both groups.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25150,"control_rate":0.19,"mde_absolute":0.01,"power_target":0.8,"treatment_rate":0.205,"control_activated":4712,"treatment_activated":5156}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Onboarding FeatureMedium

A/B Test for Onboarding CTAMedium

A/B Test for Onboarding FlowMedium

Next question

A/B Test for Onboarding Feature

Medium

Statistics & Probability

Hypothesis TestingA/B TestingStatistical Significance

Problem

Business Context

Problem Statement

Evaluate whether the new onboarding checklist increased activation rate relative to the current experience, and assess whether the observed lift is large enough to matter operationally.

Given Data

Group	Users	Activated Users	Activation Rate
Control	24,800	4,712	19.0%
Treatment	25,150	5,156	20.5%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power	0.80
Minimum detectable effect to evaluate	1.0 percentage point

Requirements

State the null and alternative hypotheses for the activation-rate comparison.
Compute the sample proportions, pooled proportion, and standard error for a two-proportion z-test.
Calculate the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the treatment-minus-control lift.
Decide whether the result is statistically significant at the 5% level.
Compare the observed lift with the 1.0 percentage point practical threshold and comment on rollout.
Briefly explain how you would think about power and whether this test appears adequately sized.

Assumptions

Random assignment was done at the user level.
Each user appears once in the experiment dataset.
No major instrumentation issues or sample-ratio mismatch were detected.
Activation is measured consistently within 7 days for both groups.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25150,"control_rate":0.19,"mde_absolute":0.01,"power_target":0.8,"treatment_rate":0.205,"control_activated":4712,"treatment_activated":5156}

Output(none)

Problem

Business Context

Problem Statement

Evaluate whether the new onboarding checklist increased activation rate relative to the current experience, and assess whether the observed lift is large enough to matter operationally.

Given Data

Group	Users	Activated Users	Activation Rate
Control	24,800	4,712	19.0%
Treatment	25,150	5,156	20.5%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Desired power	0.80
Minimum detectable effect to evaluate	1.0 percentage point

Requirements

State the null and alternative hypotheses for the activation-rate comparison.
Compute the sample proportions, pooled proportion, and standard error for a two-proportion z-test.
Calculate the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the treatment-minus-control lift.
Decide whether the result is statistically significant at the 5% level.
Compare the observed lift with the 1.0 percentage point practical threshold and comment on rollout.
Briefly explain how you would think about power and whether this test appears adequately sized.

Assumptions

Random assignment was done at the user level.
Each user appears once in the experiment dataset.
No major instrumentation issues or sample-ratio mismatch were detected.
Activation is measured consistently within 7 days for both groups.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25150,"control_rate":0.19,"mde_absolute":0.01,"power_target":0.8,"treatment_rate":0.205,"control_activated":4712,"treatment_activated":5156}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Onboarding FeatureMedium

A/B Test for Onboarding CTAMedium

A/B Test for Onboarding FlowMedium

Next question