Context

BrightAssist is deploying a customer-support generative AI model that answers billing and account questions in a fintech app. In offline evaluation, the model produces fluent responses, but the trust and safety team found cases of incorrect financial advice, policy violations, and unsafe escalation handling.

Current Performance

The Problem

Leadership wants a practical evaluation framework that measures both answer quality and safety before launch. The current metrics suggest the model is usable for simple requests but unreliable in high-risk situations, especially where escalation or factual grounding is required.

Requirements

1. Define an evaluation approach covering both quality and safety.
2. Interpret the current metrics and identify the biggest launch risks.
3. Recommend how to segment evaluation by use case, risk level, and prompt type.
4. Propose thresholding or gating rules for launch readiness.
5. Suggest concrete improvements to raise quality without weakening safety.

Constraints

• The product handles regulated financial topics.
• Human review capacity is limited to 8% of daily conversations.
• Launch cannot increase average latency above 2.5 seconds.
• A severe unsafe response is considered more costly than an unnecessary refusal.