Improve Loan Default Model Accuracy | Dataford Interview Questions

Context

LendWise uses a binary classification model to predict whether a personal loan applicant will default within 12 months. The current gradient boosted tree model was deployed 4 months ago to support underwriting decisions, but business stakeholders are concerned that overall accuracy looks acceptable while default losses remain higher than expected.

Current Performance

Metric	Validation Set	Production (last 30 days)	Baseline Logistic Regression
Accuracy	0.84	0.79	0.76
Precision	0.68	0.61	0.57
Recall	0.52	0.41	0.46
F1 Score	0.59	0.49	0.51
AUC-ROC	0.81	0.74	0.72
Default Rate	18%	21%	18%

In production over the last 30 days, the model evaluated 50,000 applications. At the current threshold, it produced: TP = 4,305, FP = 2,751, FN = 6,195, TN = 36,749.

The Problem

The Head of Risk asks: “What steps would you take to improve this model’s accuracy?” A strong answer should go beyond simply saying “collect more data” and should diagnose whether accuracy is even the right optimization target given the current error profile.

Requirements

Interpret the model’s current performance using the provided metrics.
Explain why accuracy alone may be misleading in this setting.
Identify likely causes of the production drop versus validation.
Recommend concrete steps to improve model performance, including evaluation and validation changes.
Discuss whether threshold changes, feature work, retraining, or data quality fixes should be prioritized.

Constraints

False negatives are costly because missed defaulters create direct credit losses.
False positives reduce approval volume and hurt revenue.
The risk team can only retrain the model once per month.

Problem

Context

Current Performance

Metric	Validation Set	Production (last 30 days)	Baseline Logistic Regression
Accuracy	0.84	0.79	0.76
Precision	0.68	0.61	0.57
Recall	0.52	0.41	0.46
F1 Score	0.59	0.49	0.51
AUC-ROC	0.81	0.74	0.72
Default Rate	18%	21%	18%

In production over the last 30 days, the model evaluated 50,000 applications. At the current threshold, it produced: TP = 4,305, FP = 2,751, FN = 6,195, TN = 36,749.

The Problem

Requirements

Interpret the model’s current performance using the provided metrics.
Explain why accuracy alone may be misleading in this setting.
Identify likely causes of the production drop versus validation.
Recommend concrete steps to improve model performance, including evaluation and validation changes.
Discuss whether threshold changes, feature work, retraining, or data quality fixes should be prioritized.

Constraints

False negatives are costly because missed defaulters create direct credit losses.
False positives reduce approval volume and hurt revenue.
The risk team can only retrain the model once per month.

Problem

Context

Current Performance

Metric	Validation Set	Production (last 30 days)	Baseline Logistic Regression
Accuracy	0.84	0.79	0.76
Precision	0.68	0.61	0.57
Recall	0.52	0.41	0.46
F1 Score	0.59	0.49	0.51
AUC-ROC	0.81	0.74	0.72
Default Rate	18%	21%	18%

In production over the last 30 days, the model evaluated 50,000 applications. At the current threshold, it produced: TP = 4,305, FP = 2,751, FN = 6,195, TN = 36,749.

The Problem

Requirements

Interpret the model’s current performance using the provided metrics.
Explain why accuracy alone may be misleading in this setting.
Identify likely causes of the production drop versus validation.
Recommend concrete steps to improve model performance, including evaluation and validation changes.
Discuss whether threshold changes, feature work, retraining, or data quality fixes should be prioritized.

Constraints

False negatives are costly because missed defaulters create direct credit losses.
False positives reduce approval volume and hurt revenue.
The risk team can only retrain the model once per month.

Problem

Context

Current Performance

Metric	Validation Set	Production (last 30 days)	Baseline Logistic Regression
Accuracy	0.84	0.79	0.76
Precision	0.68	0.61	0.57
Recall	0.52	0.41	0.46
F1 Score	0.59	0.49	0.51
AUC-ROC	0.81	0.74	0.72
Default Rate	18%	21%	18%

In production over the last 30 days, the model evaluated 50,000 applications. At the current threshold, it produced: TP = 4,305, FP = 2,751, FN = 6,195, TN = 36,749.

The Problem

Requirements

Interpret the model’s current performance using the provided metrics.
Explain why accuracy alone may be misleading in this setting.
Identify likely causes of the production drop versus validation.
Recommend concrete steps to improve model performance, including evaluation and validation changes.
Discuss whether threshold changes, feature work, retraining, or data quality fixes should be prioritized.

Constraints

False negatives are costly because missed defaulters create direct credit losses.
False positives reduce approval volume and hurt revenue.
The risk team can only retrain the model once per month.