Context
Intuit wants a customer-facing assistant inside QuickBooks Payments that answers merchant questions such as payment status, deposit timing, disputes, fees, failed charges, and supported payment methods. The assistant should reduce support contacts while staying tightly grounded in approved payments policy and account-specific data.
Constraints
- p95 latency: 2,500ms end-to-end
- Cost ceiling: $0.03 per request and $45K/month at 1.5M requests/month
- Hallucination ceiling: <2% on a labeled golden set for policy/account-specific factual claims
- Safety: must resist prompt injection, never reveal hidden instructions, and must not expose PII beyond what the authenticated merchant is allowed to see
- UX: if evidence is insufficient or the question is ambiguous, the assistant should ask one clarifying question or safely refuse
Available Resources
- QuickBooks Payments help-center articles, fee schedules, dispute playbooks, and merchant onboarding docs
- Structured account data APIs: recent transactions, deposits, disputes, payment methods, account status, and support case history
- A vector store and keyword index already approved by Intuit security
- Access to one high-quality LLM and one cheaper fallback model
- 1,000 historical support conversations that can be redacted and labeled for evaluation
Task
- Design the prompting and retrieval architecture for this assistant, including when to use help-center content vs. account-specific structured data.
- Write a system prompt that enforces grounded answers, safe refusal behavior, citation/use-of-evidence rules, and prompt-injection resistance.
- Define an evaluation plan first: offline metrics/datasets and online success metrics, including how you will measure hallucination, answer usefulness, and safe handling of unsupported questions.
- Estimate cost and latency for your design and explain how you would stay within budget at target volume.
- Identify the main failure modes for a payments experience and how you would detect and mitigate them in production.
