Price Increase Significance vs Tasker Fallout

Business Context

You’re a data scientist at TaskRabbit, a two-sided marketplace with ~3.5M monthly active customers and 220k active Taskers across North America. The pricing team proposes a +4% increase to the customer-facing hourly price (implemented via a higher service fee) to improve contribution margin. Ops is worried about operational fallout: if Taskers see fewer bookings (or lower effective hourly earnings), they may churn, causing longer wait times and cancellations.

A 14-day randomized experiment was run in 6 large metros. Customers were randomized at the customer-id level to see either the old price (control) or the +4% price (treatment). Taskers were not explicitly randomized; they receive jobs based on matching/availability, but the marketplace is large enough that the team assumes interference is limited.

Primary metric: booking conversion rate (a customer who views a task page and completes a booking within 24 hours).

Guardrail metric: 7-day Tasker churn (a Tasker who was active in the prior 28 days and then has zero task acceptances in the subsequent 7 days).

Given Data

Assume you can use large-sample normal approximations for proportions.

Problem Statement

You need to decide whether the +4% price change should be rolled out, balancing statistical significance with operational risk. Specifically, you must quantify:

1. whether bookings decreased significantly (revenue risk),
2. whether Tasker churn increased significantly (supply risk), and
3. what sample size would be needed to reliably detect a small churn increase before a full rollout.

Requirements

1. Bookings impact (two-proportion z-test):

   • State H_0 and H_1 for booking conversion.
   • Compute the observed conversion rates, the z-statistic, and the two-sided p-value.
   • Provide a 95% confidence interval for the difference (treatment − control).

2. Tasker churn guardrail (one-sided test):

   • Ops only cares if churn increases. State H_0 and H_1 accordingly.
   • Compute the z-statistic and the one-sided p-value.
   • Provide a 95% one-sided upper confidence bound (or equivalently a two-sided 90% CI) for the churn increase.

3. Operational decision framing:

   • Suppose the business rule is: “Proceed only if bookings do not drop by more than 1% relative and Tasker churn does not increase by more than 0.5 percentage points.” Use your CI results to assess whether the data supports proceeding.

4. Power / sample sizing:

   • Using the observed control churn rate as baseline, estimate how many Taskers per group you’d need to detect an absolute increase of +0.3 percentage points in 7-day churn with 80% power at α=0.05 (one-sided).

Assumptions and Constraints

• Randomization is at customer-id; booking events are independent across customers.
• For churn, treat Taskers in control vs treatment as approximately independent samples (acknowledge this may be imperfect due to marketplace interference).
• Use normal approximations for proportions; do not use Bayesian methods here.
• Ignore multiple testing adjustments, but comment on them in interpretation.