Business Context
StreamCart is evaluating a new recommendation widget on its product detail page. The product manager wants to compare two key metrics between control and treatment using confidence intervals rather than relying only on p-values.
Problem Statement
You need to compare the treatment and control groups on two product metrics:
- Click-through rate (CTR) on the recommendation widget, a binary metric
- Average revenue per user (ARPU), a continuous metric
Use 95% confidence intervals to determine whether the treatment meaningfully changed either metric.
Given Data
| Metric | Control | Treatment |
|---|
| Users | 18,500 | 17,900 |
| Recommendation clicks | 2,590 | 2,792 |
| CTR | 14.0% | 15.6% |
| Mean ARPU | $8.40 | $8.95 |
| ARPU standard deviation | $5.80 | $6.10 |
Assume a two-sided 95% confidence level and independent random assignment.
Requirements
- Compute the 95% confidence interval for the difference in CTR: ptreatment−pcontrol
- Compute the 95% confidence interval for the difference in ARPU: μtreatment−μcontrol
- For each metric, determine whether the interval excludes 0
- Compare the strength of evidence across the two metrics
- Explain how confidence intervals help product teams assess both statistical and practical significance
Assumptions
- Users were randomly assigned and each user appears once
- Sample sizes are large enough for normal approximations
- ARPU observations are independent across users
- Ignore multiple-testing corrections for this exercise
