Analyze A/B Test Results | Dataford Interview Questions

Shopify Checkout experiments record one binary result per visitor: 1 for a completed purchase and 0 otherwise. Given control and treatment result vectors, analyze the experiment and determine whether the treatment conversion rate is significantly different at a 5% significance level.

Implement analyze_ab_test(control, treatment).

Formal Specification

Input: Two non-empty lists of integers, where each value is either 0 or 1.
Output: A dictionary containing control and treatment sample sizes, conversion rates, absolute lift, relative lift, z-score, two-sided p-value, and a boolean significant field.
Round numeric output values to four decimal places. If the control conversion rate is zero, return None for relative lift.
Use a pooled two-proportion z-test. If the pooled standard error is zero, return z_score = 0.0 and p_value = 1.0.

Constraints

1 <= len(control), len(treatment) <= 10^6
Every vector value is either 0 or 1
The vectors may have different lengths
Use a two-sided significance threshold of 0.05

Function Signature

def analyze_ab_test(control, treatment):

Your question is Analyze A/B Test Results. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem