LLM Evaluation for Banking

How would you evaluate an LLM for factual accuracy, hallucination rate, and answer usefulness in a regulated banking setting?

Explain how you would construct representative and adversarial test data, define measurable metrics and acceptance thresholds, compare models or prompts, and validate results in production. Address citations, abstention, prompt injection, PII, auditability, human review, and cost-latency tradeoffs. Your answer should distinguish offline evaluation from online monitoring and explain how you would handle questions without a reliable ground-truth answer.

Deliverables:

1. Define the evaluation dataset, labels, metrics, and target thresholds.
2. Design offline and online evaluation procedures.
3. Describe an implementation approach and reporting format.
4. Identify major failure modes and mitigations.