A/B Test Braking Smoothness Update

Business Context

You’re a data scientist on the Autopilot/Vehicle Dynamics team at VoltRide, an EV manufacturer with ~800k connected vehicles in North America. A new over-the-air (OTA) software update changes the brake blending algorithm (regen + friction) and is expected to make deceleration feel smoother, reducing customer complaints and improving safety perception.

The team runs a randomized controlled experiment: eligible vehicles are randomly assigned to Control (old braking logic) or Treatment (new update). For each vehicle, telemetry is aggregated over a 7-day window after assignment. The primary metric is braking jerk during moderate braking events (units: m/s³). Lower jerk indicates smoother braking.

Because jerk has occasional spikes (potholes, emergency braking), the metric is defined as the 95th percentile jerk per vehicle over qualifying events (e.g., deceleration between 0.8–2.5 m/s², speed 10–80 km/h, dry road inferred from sensors).

Problem Statement

Determine whether the OTA update improved braking smoothness, defined as a reduction in mean per-vehicle 95th-percentile jerk, and quantify the size of the improvement.

Given Data

Assume per-vehicle 95th-percentile jerk values are approximately independent across vehicles. Summary statistics from the experiment are:

Additional info:

• Significance level: α = 0.05
• You care about detecting at least a 0.05 m/s³ reduction (minimum practically important difference, MPID).

Requirements

1. Define the null and alternative hypotheses (one-sided: “treatment is smoother”).
2. Compute the standard error for the difference in means using Welch’s approach (do not assume equal variances).
3. Compute the test statistic, approximate degrees of freedom, and the p-value.
4. Construct a 95% confidence interval for (μ_treatment − μ_control).
5. Interpret results in business terms: is it statistically significant, and is it practically meaningful relative to the MPID?
6. (Power/sizing) Using the observed pooled variability, estimate the per-group sample size needed for 80% power to detect a 0.05 m/s³ reduction at α=0.05 (one-sided). You may use a normal approximation.

Assumptions & Constraints

• Random assignment was implemented correctly (no targeting by geography/vehicle model beyond eligibility).
• Each vehicle contributes one aggregated value; vehicles are independent units.
• The per-vehicle metric is heavy-tailed in raw form, but the per-vehicle 95th percentile is treated as approximately normal due to aggregation; Welch’s t-test is robust to mild deviations.
• Potential confounders (weather, road conditions) should be balanced by randomization, but you should mention any residual risks.