Interview Guides

A/B Test for Ranking Model Value | Dataford Interview Questions - Dataford - Ace your Interview

A/B Test for Ranking Model Value

Easy

Statistics & Probability

Asked at 7 companies7A/B TestingStatistical SignificanceExperimentation

Also asked at

Problem

Business Context

StreamCart, a retail marketplace, built a new product-ranking model intended to improve click-through on recommendation widgets. To validate model value, the team ran a user-level A/B test comparing the current ranking model (control) against the new model (treatment).

Problem Statement

Determine whether the new ranking model creates a statistically significant improvement in recommendation click-through rate (CTR), and quantify the likely size of the lift.

Given Data

Group	Users Exposed	Users Who Clicked	Observed CTR
Control (current model)	52,400	6,131	11.7004%
Treatment (new model)	51,900	6,492	12.5087%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Randomization unit	User

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample CTR in each group and the observed absolute lift.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in CTR.
Conclude whether the model shows statistically significant value and whether the lift is practically meaningful.

Assumptions

Users were randomly assigned and appear in only one group.
Click outcome is binary at the user level.
Sample sizes are large enough for normal approximation.
No major instrumentation changes occurred during the experiment.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"control_ctr":0.117004,"treatment_n":51900,"treatment_ctr":0.125087,"control_clicks":6131,"treatment_clicks":6492}

Output(none)

Problem

Business Context

Problem Statement

Determine whether the new ranking model creates a statistically significant improvement in recommendation click-through rate (CTR), and quantify the likely size of the lift.

Given Data

Group	Users Exposed	Users Who Clicked	Observed CTR
Control (current model)	52,400	6,131	11.7004%
Treatment (new model)	51,900	6,492	12.5087%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Randomization unit	User

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample CTR in each group and the observed absolute lift.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in CTR.
Conclude whether the model shows statistically significant value and whether the lift is practically meaningful.

Assumptions

Users were randomly assigned and appear in only one group.
Click outcome is binary at the user level.
Sample sizes are large enough for normal approximation.
No major instrumentation changes occurred during the experiment.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"control_ctr":0.117004,"treatment_n":51900,"treatment_ctr":0.125087,"control_clicks":6131,"treatment_clicks":6492}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Email Campaign LiftEasy

Evaluate A/B Test Results for New FeatureMedium AA/B Test vs Observational LiftEasy

Next question

A/B Test for Ranking Model Value

Easy

Statistics & Probability

Asked at 7 companies7A/B TestingStatistical SignificanceExperimentation

Also asked at

Problem

Business Context

Problem Statement

Determine whether the new ranking model creates a statistically significant improvement in recommendation click-through rate (CTR), and quantify the likely size of the lift.

Given Data

Group	Users Exposed	Users Who Clicked	Observed CTR
Control (current model)	52,400	6,131	11.7004%
Treatment (new model)	51,900	6,492	12.5087%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Randomization unit	User

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample CTR in each group and the observed absolute lift.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in CTR.
Conclude whether the model shows statistically significant value and whether the lift is practically meaningful.

Assumptions

Users were randomly assigned and appear in only one group.
Click outcome is binary at the user level.
Sample sizes are large enough for normal approximation.
No major instrumentation changes occurred during the experiment.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"control_ctr":0.117004,"treatment_n":51900,"treatment_ctr":0.125087,"control_clicks":6131,"treatment_clicks":6492}

Output(none)

Problem

Business Context

Problem Statement

Determine whether the new ranking model creates a statistically significant improvement in recommendation click-through rate (CTR), and quantify the likely size of the lift.

Given Data

Group	Users Exposed	Users Who Clicked	Observed CTR
Control (current model)	52,400	6,131	11.7004%
Treatment (new model)	51,900	6,492	12.5087%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Randomization unit	User

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample CTR in each group and the observed absolute lift.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in CTR.
Conclude whether the model shows statistically significant value and whether the lift is practically meaningful.

Assumptions

Users were randomly assigned and appear in only one group.
Click outcome is binary at the user level.
Sample sizes are large enough for normal approximation.
No major instrumentation changes occurred during the experiment.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":52400,"control_ctr":0.117004,"treatment_n":51900,"treatment_ctr":0.125087,"control_clicks":6131,"treatment_clicks":6492}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Email Campaign LiftEasy

Evaluate A/B Test Results for New FeatureMedium AA/B Test vs Observational LiftEasy

Next question