Context

CivitasAI is preparing to deploy a text classification model that flags harmful user prompts before they reach a generative assistant. The model is a fine-tuned transformer binary classifier trained to detect harassment, self-harm encouragement, violent threats, and illegal-instruction requests.

An offline evaluation on a 50,000-example holdout set shows strong aggregate accuracy, but the Trust & Safety team is concerned that harmful misses remain too high for production launch.

Current Performance

The Problem

The model looks acceptable on top-line metrics, but recall is materially below the launch threshold. Missing harmful prompts is considered more costly than over-blocking benign prompts, especially in high-risk categories such as self-harm and explicit violence.

Task

1. Evaluate whether this model is ready for production given the current metrics.
2. Interpret the tradeoff between precision, recall, and false positive rate for this use case.
3. Identify the most important error patterns to investigate before launch.
4. Recommend how you would validate the model across harmful subcategories and user segments.
5. Propose concrete changes to improve safety performance without creating unacceptable user friction.

Constraints

1. The moderation team can manually review at most 2,500 flagged prompts per day.
2. False negatives in self-harm and violent-threat categories are treated as the highest-risk failures.
3. Product leadership will not approve launch if benign prompt block rate exceeds 5%.