Monitor Vision Model Drift

Context

ChipSight runs a computer vision model to classify memory chips on a factory inspection line as pass or defect from top-down package images. The model performed well at launch, but over the last 10 weeks the manufacturer introduced 3 new chip package variants, and QA suspects the model is missing more defects on these newer designs.

Current Performance

The Problem

Operations needs a production monitoring plan that can detect when model performance degrades as new memory chip types appear, before defect escape rates materially increase. You have delayed labels from downstream electrical testing, image embeddings, chip metadata (package type, supplier, line, week), and model confidence scores.

Task

1. Explain what the metric changes suggest about model behavior overall and on new chip variants.
2. Design a production monitoring framework, including leading indicators before labels arrive and lagging indicators after labels arrive.
3. Specify how you would segment performance to detect drift tied to new chip types.
4. Recommend alert thresholds, review cadence, and escalation criteria.
5. Propose concrete actions if degradation is confirmed.

Constraints

1. Ground-truth labels arrive 5-7 days after prediction.
2. False negatives are far more costly than false positives because missed defects can reach customers.
3. Retraining can only happen once every 2 weeks due to validation requirements.