HealthAssist AI is a customer-facing LLM that answers general wellness questions and drafts support responses for a telehealth platform. The team added a safety layer to block harmful or non-compliant outputs, but users now report that some safe questions are being refused while a smaller set of risky answers still slip through.
| Metric | Validation Set | Target | Notes |
|---|---|---|---|
| Safe response precision | 0.96 | >= 0.95 | Most allowed answers are actually safe |
| Safe response recall | 0.78 | >= 0.90 | Many safe queries are unnecessarily blocked |
| Harmful output rate | 1.8% | < 0.5% | Too many unsafe responses still reach users |
| Refusal rate | 24% | 10-15% | Over-refusal hurts usability |
| F1 score (safe vs unsafe) | 0.86 | >= 0.92 | Overall balance is weak |
| Calibration error | 0.11 | < 0.05 | Risk scores are poorly aligned to actual risk |
The model appears conservative on benign prompts but still misses some genuinely unsafe outputs. Leadership wants a practical evaluation plan that improves both safety and answer quality without making the assistant unusable.