Evaluate Financial Document Extraction Quality | Dataford Interview Questions

Your question is Evaluate Financial Document Extraction Quality. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Context

FinParse AI extracts key fields from invoices, bank statements, and loan agreements for downstream accounting workflows. After a recent rollout to enterprise customers, operations teams reported that some critical fields are being missed or extracted incorrectly, especially on scanned PDFs.

Current Performance

The system was evaluated on a labeled set of 12,000 fields across 1,500 financial documents.

Metric	Overall	Invoices	Bank Statements	Loan Agreements
Precision	0.93	0.96	0.91	0.88
Recall	0.81	0.89	0.79	0.68
F1 Score	0.87	0.92	0.84	0.77
Exact Match Rate	0.76	0.84	0.73	0.61
OCR Character Error Rate	0.058	0.031	0.064	0.089
Documents with at least 1 critical error	18.7%	11.2%	19.5%	29.8%

The Problem

Leadership wants to know whether the model is good enough for production use in partially automated review flows. The main concern is that high precision masks weak recall on legally and financially important fields such as payment due date, account number, interest rate, and maturity date.

Task

Interpret the current metrics and identify the main failure modes.
Explain whether overall performance is acceptable for financial workflows and why.
Recommend the most important additional evaluations beyond aggregate precision/recall.
Propose specific model or pipeline improvements to raise performance on the weakest document types.
Discuss how you would set acceptance criteria for production.

Constraints

False extractions can trigger incorrect downstream entries and manual rework.
Human reviewers can only check 35% of extracted documents.
Critical fields on loan agreements and bank statements must be prioritized over non-critical metadata.