Context

ShopLens built a binary classification model to predict whether a customer will click a recommended product in the app. The team has been using a logistic regression baseline and is testing a new gradient boosting model before rollout. Leadership sees slightly better top-line metrics from the new model, but the product team is concerned about whether the improvement is meaningful and whether any tradeoffs are hidden.

Current Performance

The Problem

The new model appears better on accuracy and AUC-ROC, but it detects fewer actual positive cases. You need to explain the difference between the baseline and new model in a practical evaluation setting and determine whether the new model should replace the baseline.

Requirements

1. Explain what the baseline model represents and why it is used in evaluation.
2. Compare the new model against the baseline using the metrics above.
3. Identify which metrics improved and which worsened, and what that means operationally.
4. Recommend whether to ship the new model as-is, adjust its threshold, or keep the baseline.
5. Describe what additional validation or error analysis you would run before making a final decision.

Constraints

• Recommendation slots are limited, so false positives create poor user experience.
• Missing a true click opportunity reduces engagement and revenue.
• The team can only support one production model this quarter.