Business Context
CartLane, an online retailer, is considering offering free shipping at checkout to increase conversion. Because shipping revenue would drop while order volume may rise, the decision should be based on profit per visitor, not conversion rate alone.
Problem Statement
CartLane ran a 21-day user-level A/B test. Control users saw the standard policy: customers pay shipping. Treatment users saw free shipping on all orders. Determine whether free shipping increases overall profitability per visitor and whether the result is statistically significant.
Given Data
| Group | Visitors | Orders | Conversion Rate | Average Merchandise Revenue per Order | Variable Product Cost per Order | Shipping Fee Charged to Customer | Shipping Cost to Company |
|---|
| Control | 50,000 | 3,900 | 7.80% | $82.00 | $49.20 | $8.00 | $10.50 |
| Treatment | 50,000 | 4,350 | 8.70% | $80.00 | $48.00 | $0.00 | $10.80 |
Assume profit is measured as:
profit per order=merchandise revenue−product cost+shipping fee collected−shipping cost
Requirements
- Compute profit per order for each group.
- Compute expected profit per visitor for control and treatment.
- State the null and alternative hypotheses for profit per visitor.
- Model per-visitor profit as a Bernoulli mixture and compute the standard error of the difference in mean profit per visitor.
- Calculate the z-statistic, two-sided p-value, and a 95% confidence interval for the treatment lift.
- Conclude whether CartLane should roll out free shipping based on both statistical and practical significance.
Assumptions
- Users were randomly assigned 50/50 and counted once.
- Order-level profit within each group is constant at the group average shown above.
- Independence across visitors is reasonable, so the large-sample normal approximation applies.
- No major inventory or marketing changes occurred during the experiment.
