Optimize On-Vehicle Pedestrian Classifier Latency

Business Context

You’re interviewing for an ML engineering role on the on-vehicle perception team at a major OEM building an L2+/L3 ADAS stack. The current camera-based pedestrian detection subsystem is accurate in offline evaluation, but it misses real-time deadlines on the production ECU, causing frame drops and delayed braking decisions. Safety and regulatory scrutiny are high: a regression in pedestrian recall can create unacceptable risk.

Your task is to propose and implement a model optimization plan that ensures real-time performance on a vehicle while maintaining detection quality.

Dataset

You are given a curated dataset derived from fleet logs:

• Source: Forward-facing 1920×1080 RGB camera (30 FPS), diverse weather/lighting, urban/suburban.
• Labeling: Pedestrian bounding boxes + “near-miss” hard negatives (e.g., poles, cyclists, strollers).
• Training objective for this interview: simplify to binary classification on cropped proposals (pedestrian vs non-pedestrian), representing the second stage of a detector.

• Size: 12.5M crops (approx. 4 TB raw), collected from ~80K drives
• Class balance: ~2.5% positive (pedestrian), 97.5% negative
• Missing data: ~8% missing metadata due to sensor dropouts; images always present

Success Criteria

You must meet both quality and runtime targets:

1. Quality (offline test set)
   • Recall (pedestrian) ≥ 0.92 at FPR ≤ 1% (safety prefers recall)
   • AUC-PR ≥ 0.70 (imbalanced data)
2. On-vehicle runtime (ECU constraints)
   • End-to-end model inference p95 ≤ 12 ms per crop on target hardware (ARM CPU + mobile GPU / NPU)
   • Model size ≤ 25 MB (flash + OTA constraints)
   • Peak RAM during inference ≤ 200 MB

Constraints & Real-World Details

• The ECU runs multiple real-time tasks; you cannot assume full GPU availability.
• The model must be robust to distribution shift (night/rain/construction zones).
• You must support A/B shadow deployment: run optimized model alongside baseline and compare disagreement rates.
• You cannot change the upstream proposal generator in this interview; focus on optimizing the classifier.

Deliverables

1. Optimization strategy: Describe a concrete plan to achieve the latency/memory targets (e.g., architecture changes, quantization, pruning, distillation, batching strategy, operator fusion).
2. Training approach: How you handle class imbalance and hard negatives, and how you avoid overfitting to easy backgrounds.
3. Evaluation plan: Metrics, threshold selection for the recall/FPR constraint, and how you would validate on-vehicle performance (profiling + correctness checks).
4. Implementation: Provide code that trains a baseline model, then applies at least one optimization technique (e.g., quantization-aware training or post-training quantization) and evaluates the quality impact.
5. Production rollout: How you would monitor regressions (latency + safety metrics) and roll back safely.