Scenario

You are building an internal finance research copilot for analysts who need fast answers about issuers, sectors, earnings, and market events. The assistant must synthesize information from internal research notes, approved market commentary, earnings-call transcripts, and regulatory filings, then produce concise, source-backed answers for time-sensitive workflows. Usage is expected to reach a few thousand analyst queries per day, with heavy spikes around earnings releases and macro events. Analysts will trust the tool only if it cites evidence and avoids confident but unsupported statements.

Constraints

• p95 latency must stay under 2,500ms for a standard question
• Cost ceiling: $25K/month at 80K queries
• Unsupported factual claims must stay below 2% on a labeled evaluation set
• Every material factual statement must be traceable to retrieved sources
• The system must resist prompt injection from retrieved documents and user input

Available Resources

• Internal research notes, approved commentary, earnings transcripts, and filings with metadata
• An approved LLM API, embedding model, and managed vector search
• Search logs and analyst feedback from the current keyword-based workflow
• Capacity for 400 manually labeled evaluation questions per quarter

Question

How would you build this assistant so it improves analyst productivity without increasing hallucination or compliance risk, and how would you evaluate it before launch and monitor it in production?