Validate Batch Quality Prediction Model

Context

Avid Bioservices deployed a binary classification model to predict whether a manufacturing batch will fail final quality review before release. The model is used in the process development workflow to prioritize additional investigation on high-risk batches, but operations leadership is concerned that the current validation approach relies too heavily on overall accuracy.

Current Performance

The Problem

The model appears stronger on accuracy and AUC-ROC, but it is catching fewer truly failing batches than the previous baseline. In Avid Bioservices manufacturing, a missed failing batch can create release delays, rework, and client risk, while too many false alarms increase investigation workload.

Requirements

1. Explain how you would validate this model beyond a single metric like accuracy.
2. Interpret whether the current model is actually better than the baseline for this use case.
3. Use the metric table and confusion matrix to identify the main risk in deployment.
4. Recommend what additional validation checks you would run before approving the model.
5. Propose specific improvements to the model or thresholding strategy.

Constraints

• Only 12% of batches fail, so the data is imbalanced.
• Quality investigations are expensive, but missed failures are more costly than unnecessary reviews.
• The model must be interpretable enough for manufacturing and quality stakeholders to trust its outputs.