Business Context

You’re interviewing for an ML engineering role on the autonomy team at WayDrive, a ride-hailing company operating a geo-fenced Level-4 autonomous fleet in Phoenix and Austin. The fleet drives ~2.5 million miles/week and must satisfy strict internal safety gates before any policy update can be deployed. A recent incident review showed that the current rule-based planner is overly conservative during merges and lane changes, causing 7–12% longer ETAs and frequent “handoff-to-remote-assist” events that cost the business ~$18 per intervention.

Your task is to propose and implement a reinforcement learning (RL) approach to improve highway driving behavior (lane keeping, lane changes, merges) while meeting safety and latency constraints.

Dataset / Environment

You are given an offline dataset collected from a mixture of human-driven and autonomy-driven trajectories, plus a lightweight simulator for evaluation.

Target behavior: highway autonomy at 25–75 mph including lane changes and merges.

Success Criteria (what “good” looks like)

You must propose an RL solution that, in simulator evaluation on a held-out scenario suite:

1. Safety: Collision rate < 0.02 per 1,000 miles; near-collision (TTC<1.0s) < 0.5 per 1,000 miles.
2. Efficiency: Mean speed relative to traffic improves by ≥ 5% vs baseline planner without increasing safety events.
3. Comfort: Hard-brake events < 0.8 per 1,000 miles; jerk p95 decreases by ≥ 10%.
4. Stability: Policy does not oscillate between lanes; lane-change abort rate < 3%.

Constraints

• Deployment latency: inference budget ≤ 20 ms on an automotive-grade GPU (e.g., Orin-class).
• Safety/regulatory: must include a safety layer (constraint handling) and an offline evaluation plan before any on-road test.
• Offline-first: you cannot do unlimited online exploration; training should primarily leverage the offline dataset.
• Interpretability: provide actionable diagnostics (what scenarios fail, why).

Deliverables

1. Explain reinforcement learning in your own words using the MDP framing (state, action, reward, transition, policy) and map each element to this autonomous driving problem.
2. Choose an RL algorithm suitable for continuous control and offline data (e.g., IQL/CQL/TD3+BC) and justify the choice.
3. Define a reward function (or reward model) that encodes safety, comfort, and progress; discuss pitfalls like reward hacking.
4. Provide a training + evaluation plan, including scenario-based validation, off-policy evaluation, and how you’d detect distribution shift.
5. Provide implementation-quality code for a minimal offline RL training loop and evaluation metrics computation.