Context

Streamly uses a binary classification model to predict whether a subscriber will cancel in the next 30 days. The model outputs a churn probability, and the retention team offers a $12 discount to users predicted as high risk. The current threshold of 0.50 was chosen during development, but finance believes the team may be overspending on discounts while still missing too many churners.

Current Performance

Validation set size: 100,000 users, with an observed 12% churn rate.

Additional model metrics: AUC-ROC = 0.81, log loss = 0.41, calibration slope = 0.88.

The Problem

You need to recommend an operating threshold for converting probabilities into churn labels. The answer should reflect business costs, not just maximize a single metric.

Requirements

1. Compare the threshold options using the metrics above.
2. Explain which threshold you would choose under different business objectives.
3. Quantify the tradeoff between false positives and false negatives.
4. Comment on whether calibration quality affects threshold selection.
5. Recommend how you would validate the chosen threshold before rollout.

Constraints

• Each retention offer costs $12.
• A prevented churn is worth $95 in expected gross margin.
• The team can contact at most 20,000 users per month.
• Product leadership wants recall above 60% if ROI remains positive.