Business Context

FinFlow, a document automation platform, processes invoices, receipts, and tax forms submitted by enterprise customers. Standard OCR works on clean scans, but accuracy drops on multi-column layouts, tables, stamps, handwritten notes, and low-quality mobile photos, so the team wants a multimodal model that uses text, layout, and image signals together.

Data

• Volume: 2.4M document pages with OCR tokens and bounding boxes; 180K pages have field-level ground truth
• Document types: invoices (52%), receipts (21%), forms (17%), statements (10%)
• Text length: 20-2,500 OCR tokens per page; median 310
• Language: English (78%), Spanish (14%), French (8%)
• Labels: key-value extraction targets such as invoice_number, vendor_name, invoice_date, total_amount, tax_amount, and line_item_total
• Class skew: some fields appear on >95% of pages, others on <40%

Success Criteria

A production-ready system should achieve entity-level F1 >= 0.90 on core fields, page inference latency < 400 ms on an A10 GPU, and remain robust to rotated scans, noisy OCR, and variable templates.

Constraints

• PII must remain in a private VPC
• The model must support weekly retraining with newly labeled documents
• The solution should degrade gracefully when OCR quality is poor

Requirements

1. Build a multimodal OCR post-processing pipeline that combines OCR text, token coordinates, and page image features.
2. Fine-tune a layout-aware transformer for token-level field extraction.
3. Define preprocessing for OCR normalization, bounding-box scaling, and long-page truncation/chunking.
4. Show a modern Python implementation using Hugging Face Transformers and a realistic training loop.
5. Explain how you would evaluate extraction quality across document types, languages, and OCR noise levels.
6. Describe fallback behavior for pages with missing tokens, bad scans, or unseen templates.