Evaluate Coaching Match Quality Model

Context

BetterUp uses a binary classification model to predict whether a newly proposed coach-member match will lead to a successful first 30 days, defined as the member attending the first session and giving a post-session rating of 4 or 5. The model is used in the matching workflow inside BetterUp Care to prioritize recommended coach options.

A new version of the model was deployed last month. Leadership sees slightly higher overall accuracy, but member complaints about poor-fit recommendations have increased, especially for enterprise members in their first week on the platform.

Current Performance

The Problem

The current model is more conservative: it recommends fewer matches as likely successful, and those recommendations are more often correct, but it misses many matches that would have succeeded. You need to assess whether the new model is actually better for BetterUp's matching experience.

Requirements

1. Interpret what the metric changes imply about model behavior.
2. Explain why higher accuracy does not necessarily mean a better model here.
3. Use the confusion-matrix implications to discuss business impact.
4. Recommend which metrics should be primary for this use case and why.
5. Propose specific next steps to improve evaluation and model performance.

Constraints

• BetterUp wants to avoid showing too few viable coach options to new members.
• False positives create some member friction, but false negatives reduce match coverage and can delay time-to-first-session.
• The matching team can only retrain and redeploy once every two weeks.