Choose Metrics for Loan Default Model | Dataford Interview Questions

Context

LendWise uses a binary classification model to predict whether a small-business loan applicant will default within 12 months. The model is used to approve, decline, or manually review applications, but leadership is concerned that the team is reporting only accuracy while portfolio losses are rising.

Current Performance

Metric	Validation Set	Last Month in Production
Accuracy	0.91	0.90
Precision	0.58	0.55
Recall	0.34	0.29
F1 Score	0.43	0.38
AUC-ROC	0.81	0.79
Log Loss	0.29	0.33
Default Rate	0.11	0.13
Manual Review Rate	7.5%	6.8%
Monthly charge-off loss	$1.9M	$2.6M

The Problem

The model still looks strong on accuracy because defaults are relatively rare, but it is missing many risky applicants. Product, risk, and operations teams disagree on which metric should define success: risk wants higher recall on defaults, sales wants fewer false declines, and operations wants to keep manual review under capacity.

Task

Identify which metrics should be primary, secondary, and monitoring metrics for this project.
Explain why accuracy is insufficient in this setting.
Interpret the current metric pattern and what it implies about model behavior.
Recommend how you would choose an operating threshold given business trade-offs.
Propose evaluation and validation steps to ensure the chosen metrics align with business outcomes.

Constraints

Manual review team can handle at most 8% of applications.
A false negative costs about $18,000 in expected losses.
A false positive decline costs about $1,200 in lost profit and customer friction.
Regulators require consistent performance monitoring across applicant segments.

Problem

Context

Current Performance

Metric	Validation Set	Last Month in Production
Accuracy	0.91	0.90
Precision	0.58	0.55
Recall	0.34	0.29
F1 Score	0.43	0.38
AUC-ROC	0.81	0.79
Log Loss	0.29	0.33
Default Rate	0.11	0.13
Manual Review Rate	7.5%	6.8%
Monthly charge-off loss	$1.9M	$2.6M

The Problem

Task

Identify which metrics should be primary, secondary, and monitoring metrics for this project.
Explain why accuracy is insufficient in this setting.
Interpret the current metric pattern and what it implies about model behavior.
Recommend how you would choose an operating threshold given business trade-offs.
Propose evaluation and validation steps to ensure the chosen metrics align with business outcomes.

Constraints

Manual review team can handle at most 8% of applications.
A false negative costs about $18,000 in expected losses.
A false positive decline costs about $1,200 in lost profit and customer friction.
Regulators require consistent performance monitoring across applicant segments.

Problem

Context

Current Performance

Metric	Validation Set	Last Month in Production
Accuracy	0.91	0.90
Precision	0.58	0.55
Recall	0.34	0.29
F1 Score	0.43	0.38
AUC-ROC	0.81	0.79
Log Loss	0.29	0.33
Default Rate	0.11	0.13
Manual Review Rate	7.5%	6.8%
Monthly charge-off loss	$1.9M	$2.6M

The Problem

Task

Identify which metrics should be primary, secondary, and monitoring metrics for this project.
Explain why accuracy is insufficient in this setting.
Interpret the current metric pattern and what it implies about model behavior.
Recommend how you would choose an operating threshold given business trade-offs.
Propose evaluation and validation steps to ensure the chosen metrics align with business outcomes.

Constraints

Manual review team can handle at most 8% of applications.
A false negative costs about $18,000 in expected losses.
A false positive decline costs about $1,200 in lost profit and customer friction.
Regulators require consistent performance monitoring across applicant segments.

Problem

Context

Current Performance

Metric	Validation Set	Last Month in Production
Accuracy	0.91	0.90
Precision	0.58	0.55
Recall	0.34	0.29
F1 Score	0.43	0.38
AUC-ROC	0.81	0.79
Log Loss	0.29	0.33
Default Rate	0.11	0.13
Manual Review Rate	7.5%	6.8%
Monthly charge-off loss	$1.9M	$2.6M

The Problem

Task

Identify which metrics should be primary, secondary, and monitoring metrics for this project.
Explain why accuracy is insufficient in this setting.
Interpret the current metric pattern and what it implies about model behavior.
Recommend how you would choose an operating threshold given business trade-offs.
Propose evaluation and validation steps to ensure the chosen metrics align with business outcomes.

Constraints

Manual review team can handle at most 8% of applications.
A false negative costs about $18,000 in expected losses.
A false positive decline costs about $1,200 in lost profit and customer friction.
Regulators require consistent performance monitoring across applicant segments.