Interview Guides

Trustworthiness Check for Signup A/B Test | Dataford Interview Questions - Dataford - Ace your Interview

Trustworthiness Check for Signup A/B Test

Medium

Statistics & Probability

Asked at 1 company1A/B TestingConfidence IntervalsP-Values

Also asked at

Problem

Business Context

StreamCart, a subscription video platform, tested a new signup page intended to increase paid trial starts. The product manager sees a positive lift, but before rollout wants to know whether the result is trustworthy.

Problem Statement

Assess whether the observed A/B test result is statistically credible and identify the main checks you would use to judge trustworthiness. Use the data below to evaluate statistical significance, confidence interval, and sample ratio mismatch.

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Trial starts	6,026	5,950
Conversion rate	11.50%	12.50%

Additional test settings:

Parameter	Value
Planned traffic split	50% / 50%
Test duration	14 days
Significance level	0.05

Requirements

State the null and alternative hypotheses for conversion rate.
Run a two-proportion z-test and compute the p-value.
Compute a 95% confidence interval for the treatment minus control lift.
Check whether the traffic allocation suggests a sample ratio mismatch (SRM) using a chi-square goodness-of-fit test.
Based on the numbers, explain whether the result is trustworthy and what else you would inspect before shipping.

Assumptions

User-level randomization was intended to be independent.
Each user is counted once.
Normal approximation is acceptable because both groups have large sample sizes and many conversions/non-conversions.
Ignore secondary metrics for the numerical calculations, but discuss them qualitatively in the interpretation.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"treatment_n":47600,"control_rate":0.115,"treatment_rate":0.125,"test_duration_days":14,"control_conversions":6026,"planned_control_share":0.5,"treatment_conversions":5950,"planned_treatment_share":0.5}

Output(none)

Problem

Business Context

Problem Statement

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Trial starts	6,026	5,950
Conversion rate	11.50%	12.50%

Additional test settings:

Parameter	Value
Planned traffic split	50% / 50%
Test duration	14 days
Significance level	0.05

Requirements

State the null and alternative hypotheses for conversion rate.
Run a two-proportion z-test and compute the p-value.
Compute a 95% confidence interval for the treatment minus control lift.
Check whether the traffic allocation suggests a sample ratio mismatch (SRM) using a chi-square goodness-of-fit test.
Based on the numbers, explain whether the result is trustworthy and what else you would inspect before shipping.

Assumptions

User-level randomization was intended to be independent.
Each user is counted once.
Normal approximation is acceptable because both groups have large sample sizes and many conversions/non-conversions.
Ignore secondary metrics for the numerical calculations, but discuss them qualitatively in the interpretation.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"treatment_n":47600,"control_rate":0.115,"treatment_rate":0.125,"test_duration_days":14,"control_conversions":6026,"planned_control_share":0.5,"treatment_conversions":5950,"planned_treatment_share":0.5}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Validate A/B Test for Signup FlowMedium

A/B Test Validation for Signup RedesignEasy

Conversion Significance for Signup RedesignEasy

Next question

Trustworthiness Check for Signup A/B Test

Medium

Statistics & Probability

Asked at 1 company1A/B TestingConfidence IntervalsP-Values

Also asked at

Problem

Business Context

Problem Statement

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Trial starts	6,026	5,950
Conversion rate	11.50%	12.50%

Additional test settings:

Parameter	Value
Planned traffic split	50% / 50%
Test duration	14 days
Significance level	0.05

Requirements

State the null and alternative hypotheses for conversion rate.
Run a two-proportion z-test and compute the p-value.
Compute a 95% confidence interval for the treatment minus control lift.
Check whether the traffic allocation suggests a sample ratio mismatch (SRM) using a chi-square goodness-of-fit test.
Based on the numbers, explain whether the result is trustworthy and what else you would inspect before shipping.

Assumptions

User-level randomization was intended to be independent.
Each user is counted once.
Normal approximation is acceptable because both groups have large sample sizes and many conversions/non-conversions.
Ignore secondary metrics for the numerical calculations, but discuss them qualitatively in the interpretation.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"treatment_n":47600,"control_rate":0.115,"treatment_rate":0.125,"test_duration_days":14,"control_conversions":6026,"planned_control_share":0.5,"treatment_conversions":5950,"planned_treatment_share":0.5}

Output(none)

Problem

Business Context

Problem Statement

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Trial starts	6,026	5,950
Conversion rate	11.50%	12.50%

Additional test settings:

Parameter	Value
Planned traffic split	50% / 50%
Test duration	14 days
Significance level	0.05

Requirements

State the null and alternative hypotheses for conversion rate.
Run a two-proportion z-test and compute the p-value.
Compute a 95% confidence interval for the treatment minus control lift.
Check whether the traffic allocation suggests a sample ratio mismatch (SRM) using a chi-square goodness-of-fit test.
Based on the numbers, explain whether the result is trustworthy and what else you would inspect before shipping.

Assumptions

User-level randomization was intended to be independent.
Each user is counted once.
Normal approximation is acceptable because both groups have large sample sizes and many conversions/non-conversions.
Ignore secondary metrics for the numerical calculations, but discuss them qualitatively in the interpretation.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"treatment_n":47600,"control_rate":0.115,"treatment_rate":0.125,"test_duration_days":14,"control_conversions":6026,"planned_control_share":0.5,"treatment_conversions":5950,"planned_treatment_share":0.5}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Validate A/B Test for Signup FlowMedium

A/B Test Validation for Signup RedesignEasy

Conversion Significance for Signup RedesignEasy

Next question