Business Context
PayPilot is a regulated fintech platform (US + EU) with 18M monthly active users and a customer support operation handling 220K tickets/day across chat and email. Support agents use an internal knowledge base (KB) spanning product docs, incident postmortems, and policy memos. Today, resolution time is inconsistent (P50: 2.1 hours, P95: 31 hours), and compliance escalations (e.g., disputes, fraud, GDPR requests) are frequently misrouted. Leadership wants an enterprise-ready AI agent that can answer customers, draft agent responses, and automatically route high-risk requests—while providing strong auditability and measurable quality.
The agent must incorporate retrieval (RAG), orchestration, policy-based routing, tool invocation, an evaluation harness, and lifecycle observability suitable for production.
Data Characteristics
- User messages: ~20–1,200 tokens (median ~180). English (85%), Spanish (10%), French (5%). Heavy domain vocabulary: chargebacks, ACH returns, KYC, 3DS, MCC codes.
- Knowledge base: ~1.8M documents (PDF/HTML/Markdown). Many are versioned; policy memos supersede older guidance. Some content is confidential (employee-only).
- Labels / signals available:
- Historical ticket metadata: queue (Billing, Fraud, Account Access, Disputes, GDPR), resolution outcome, escalation flags.
- Incident timelines and affected product components.
- A small curated set (25K) of “gold” QA pairs with citations.
- Risk distribution: ~2% fraud-related, ~4% dispute/chargeback, ~1% GDPR/DSAR, but these categories carry high regulatory and financial risk.
Success Criteria
- Containment: 35% of tickets resolved without human intervention for low-risk categories.
- Safety/compliance: ≥99% recall on “must-escalate” cases (fraud, disputes, GDPR/DSAR, account takeover signals).
- Answer quality: ≥0.80 citation-supported correctness on the gold QA set; hallucination rate <1% (answers with no supporting evidence).
- Latency: P95 < 2.5s end-to-end for chat; < 6s for email drafting.
- Auditability: Every response must log retrieved sources, policy decisions, tool calls, and model versions.
Constraints
- Regulatory: SOC2 + GDPR. No training on raw user PII without explicit governance. Strict data retention and access controls.
- Security: KB has ACLs; retrieval must be permission-aware.
- Reliability: Must degrade gracefully if retrieval or tools fail (fallback to safe escalation).
- Cost: Target <$0.015 per chat turn on average.
Requirements (Deliverables)
- Agent architecture: Propose a modular design including retrieval, policy router, orchestrator, and tool layer.
- Retrieval design: Chunking strategy, embedding model choice, hybrid retrieval (BM25 + vectors), reranking, and ACL filtering.
- Policy-based routing: Define a routing policy that classifies requests into: Self-serve, Agent-assist, Must-escalate. Include multilingual handling.
- Tool invocation: Implement at least three tools (e.g., account status lookup, dispute status, incident status) with safe-guards and structured outputs.
- Evaluation harness: Offline evaluation covering retrieval quality, answer correctness with citations, routing recall for high-risk cases, and regression tests.
- Lifecycle observability: Specify what to log/trace/monitor (prompt+retrieval traces, tool errors, drift, cost, latency) and how to set alerts.
- Rollout plan: Canary + shadow deployment, human-in-the-loop review, and rollback criteria.
Your answer should include concrete implementation details and justify trade-offs (e.g., reranker latency vs quality, strict escalation vs containment).
