
StreamCart, a subscription video platform, tested a new signup page intended to increase paid trial starts. The product manager sees a positive lift, but before rollout wants to know whether the result is trustworthy.
Assess whether the observed A/B test result is statistically credible and identify the main checks you would use to judge trustworthiness. Use the data below to evaluate statistical significance, confidence interval, and sample ratio mismatch.
| Metric | Control | Treatment |
|---|---|---|
| Assigned users | 52,400 | 47,600 |
| Trial starts | 6,026 | 5,950 |
| Conversion rate | 11.50% | 12.50% |
Additional test settings:
| Parameter | Value |
|---|---|
| Planned traffic split | 50% / 50% |
| Test duration | 14 days |
| Significance level | 0.05 |
{"alpha":0.05,"control_n":52400,"treatment_n":47600,"control_rate":0.115,"treatment_rate":0.125,"test_duration_days":14,"control_conversions":6026,"planned_control_share":0.5,"treatment_conversions":5950,"planned_treatment_share":0.5}Output(none)