Interview Guides

A/B Test for Onboarding CTA | Dataford Interview Questions - Dataford - Ace your Interview

A/B Test for Onboarding CTA

Medium

Statistics & Probability

Asked at 6 companies6A/B TestingConfidence IntervalsExperimentation

Also asked at

Problem

Business Context

PulseBoard, a B2B analytics SaaS product, tested a new onboarding call-to-action intended to get more new signups to connect their first data source. Product leadership wants to know whether the experiment shows a real improvement and whether the lift is large enough to justify rollout.

Problem Statement

You led this product experiment and now need to analyze the result. Determine whether the new onboarding CTA improved 7-day activation rate relative to the existing experience.

Given Data

Group	Users	Activated within 7 days	Activation Rate
Control (old CTA)	18,420	7,184	39.00%
Treatment (new CTA)	18,105	7,430	41.04%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Experiment duration	21 days

Requirements

State the null and alternative hypotheses.
Compute the sample activation rates and the absolute lift.
Calculate the pooled proportion and standard error for a two-proportion z-test.
Compute the z-statistic and p-value.
Construct a 95% confidence interval for the difference in activation rates.
Decide whether to roll out the new CTA, using both statistical and practical significance.

Assumptions

Users were randomly assigned at signup.
Each signup appears once in the analysis.
No major traffic-source mix shift occurred during the 21-day test.
Activation is binary: a user either connected a data source within 7 days or did not.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18420,"treatment_n":18105,"control_rate":0.39,"treatment_rate":0.41038387185970726,"experiment_days":21,"control_activated":7184,"treatment_activated":7430}

Output(none)

Problem

Business Context

Problem Statement

You led this product experiment and now need to analyze the result. Determine whether the new onboarding CTA improved 7-day activation rate relative to the existing experience.

Given Data

Group	Users	Activated within 7 days	Activation Rate
Control (old CTA)	18,420	7,184	39.00%
Treatment (new CTA)	18,105	7,430	41.04%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Experiment duration	21 days

Requirements

State the null and alternative hypotheses.
Compute the sample activation rates and the absolute lift.
Calculate the pooled proportion and standard error for a two-proportion z-test.
Compute the z-statistic and p-value.
Construct a 95% confidence interval for the difference in activation rates.
Decide whether to roll out the new CTA, using both statistical and practical significance.

Assumptions

Users were randomly assigned at signup.
Each signup appears once in the analysis.
No major traffic-source mix shift occurred during the 21-day test.
Activation is binary: a user either connected a data source within 7 days or did not.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18420,"treatment_n":18105,"control_rate":0.39,"treatment_rate":0.41038387185970726,"experiment_days":21,"control_activated":7184,"treatment_activated":7430}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Onboarding FeatureMedium A/B Test for Onboarding FeatureMedium

Retention A/B Test for OnboardingMedium

Next question

A/B Test for Onboarding CTA

Medium

Statistics & Probability

Asked at 6 companies6A/B TestingConfidence IntervalsExperimentation

Also asked at

Problem

Business Context

Problem Statement

You led this product experiment and now need to analyze the result. Determine whether the new onboarding CTA improved 7-day activation rate relative to the existing experience.

Given Data

Group	Users	Activated within 7 days	Activation Rate
Control (old CTA)	18,420	7,184	39.00%
Treatment (new CTA)	18,105	7,430	41.04%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Experiment duration	21 days

Requirements

State the null and alternative hypotheses.
Compute the sample activation rates and the absolute lift.
Calculate the pooled proportion and standard error for a two-proportion z-test.
Compute the z-statistic and p-value.
Construct a 95% confidence interval for the difference in activation rates.
Decide whether to roll out the new CTA, using both statistical and practical significance.

Assumptions

Users were randomly assigned at signup.
Each signup appears once in the analysis.
No major traffic-source mix shift occurred during the 21-day test.
Activation is binary: a user either connected a data source within 7 days or did not.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18420,"treatment_n":18105,"control_rate":0.39,"treatment_rate":0.41038387185970726,"experiment_days":21,"control_activated":7184,"treatment_activated":7430}

Output(none)

Problem

Business Context

Problem Statement

You led this product experiment and now need to analyze the result. Determine whether the new onboarding CTA improved 7-day activation rate relative to the existing experience.

Given Data

Group	Users	Activated within 7 days	Activation Rate
Control (old CTA)	18,420	7,184	39.00%
Treatment (new CTA)	18,105	7,430	41.04%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Experiment duration	21 days

Requirements

State the null and alternative hypotheses.
Compute the sample activation rates and the absolute lift.
Calculate the pooled proportion and standard error for a two-proportion z-test.
Compute the z-statistic and p-value.
Construct a 95% confidence interval for the difference in activation rates.
Decide whether to roll out the new CTA, using both statistical and practical significance.

Assumptions

Users were randomly assigned at signup.
Each signup appears once in the analysis.
No major traffic-source mix shift occurred during the 21-day test.
Activation is binary: a user either connected a data source within 7 days or did not.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":18420,"treatment_n":18105,"control_rate":0.39,"treatment_rate":0.41038387185970726,"experiment_days":21,"control_activated":7184,"treatment_activated":7430}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Onboarding FeatureMedium A/B Test for Onboarding FeatureMedium

Retention A/B Test for OnboardingMedium

Next question