Evaluate Model Metrics for Customer Churn Prediction

Context

ChurnGuard, a SaaS company, recently deployed a logistic regression model to predict customer churn based on user behavior and engagement metrics. After three months in production, the model's performance has raised concerns as it consistently flags a low number of churn cases.

Current Performance

The Problem

Despite maintaining a high precision of 88%, the model's recall has dropped from 40% to 35%, resulting in a significant number of missed churn cases. This decline in recall raises alarms for the customer success team, as they rely on accurate predictions to proactively engage customers at risk of leaving.

Task

1. Diagnose the reasons for the drop in recall and suggest at least 3 hypotheses.
2. Propose a systematic error analysis approach to validate these hypotheses.
3. Recommend 3-4 specific model improvements and their expected impact.
4. Discuss the trade-offs involved in improving recall versus maintaining precision.

Constraints

• The customer success team has limited resources for outreach (1,000 proactive engagements per day).
• High churn costs approximately $500 per lost customer, necessitating a focus on effective targeting.