Context

Microsoft Teams uses a binary classifier to detect abusive chat messages and route high-risk messages to automated enforcement or human review. The model is a fine-tuned gradient boosting classifier that outputs a probability score, but the team is debating which decision threshold to use before rolling out to all enterprise tenants.

Current Performance

Validation set size: 200,000 messages with 4,000 abusive messages (2.0% prevalence).

Additional model metrics on the same validation set:

The Problem

The Trust & Safety team wants high recall to reduce harmful content exposure, while enterprise customers are sensitive to false positives because incorrect enforcement can block legitimate workplace communication. You need to recommend an operating threshold and explain the tradeoff.

Requirements

1. Recommend the best threshold for general deployment and justify it using the table.
2. Explain how business costs of false positives vs. false negatives should influence threshold choice.
3. Describe how you would validate whether the scores are well calibrated before thresholding.
4. Propose a thresholding strategy for different enforcement actions (auto-hide, human review, allow).
5. Identify what additional offline or online evaluation you would run before launch.

Constraints

• Human moderators can review at most 4,000 messages/day.
• False positives on executive or customer-facing chats are considered high severity.
• Missing abusive content is costly because repeated exposure increases user complaints and tenant churn.