SafeMarket uses a text classification model to flag user-generated product listings for manual safety review. The model decides which listings enter a limited human review queue, and leadership is concerned that the current setup may be missing too many unsafe listings while also consuming reviewer capacity.
The team evaluated one month of labeled listings after deployment. Unsafe content includes prohibited medical claims, dangerous product instructions, and regulated items requiring removal.
| Metric | Current Model | Previous Threshold Setting |
|---|---|---|
| Precision | 0.91 | 0.78 |
| Recall | 0.54 | 0.72 |
| F1 Score | 0.68 | 0.75 |
| AUC-ROC | 0.89 | 0.89 |
| Listings flagged for review | 4,800 | 7,900 |
| Confirmed unsafe listings | 4,000 | 4,000 |
| Unsafe listings caught | 2,160 | 2,880 |
| Unsafe listings missed | 1,840 | 1,120 |
The model is highly precise, but recall is materially lower than the previous operating point. Reviewers report that most flagged listings are truly unsafe, but policy teams are escalating missed violations found through user reports and audits.