Context
Meta's client-facing teams want an internal assistant that answers questions about Meta advertising products, including Ads Manager, Meta Pixel, Conversions API, Advantage+ shopping campaigns, campaign setup, measurement, and policy-related guidance. The assistant should help sales, solutions architects, and support teams respond quickly while grounding answers in approved Meta documentation.
Constraints
- p95 latency: 2,500ms for a standard question
- Cost ceiling: $25K/month at 40K queries/day
- Hallucination ceiling: <2% on a labeled evaluation set, where any unsupported factual claim counts as a hallucination
- Must provide source citations for product facts, setup steps, and policy statements
- Must resist prompt injection from retrieved content and user messages
- Must respect document permissions and avoid exposing restricted internal or client-specific information
Available Data / Models
- Approved corpus of Meta advertising documentation: product guides, help center articles, API docs, policy docs, release notes, troubleshooting playbooks, and internal enablement content
- Metadata per document: product area, locale, publish date, owner team, audience, permission tier
- A vector database with metadata filtering and BM25 support
- Access to an approved LLM and embedding model via the OpenAI or Anthropic SDK
- 500 historically answered support and solutions questions that can seed a golden evaluation set
Deliverables
- Design an end-to-end RAG workflow for ingestion, chunking, indexing, retrieval, reranking, answer generation, and citation verification.
- Define the evaluation plan first: offline metrics, adversarial tests, and online success metrics, including how you measure hallucination and retrieval quality.
- Propose a system prompt and runtime guardrails that enforce grounded answers, refusal behavior, and prompt-injection resistance.
- Estimate cost and latency at target volume, and explain what changes you would make if you miss either budget.
- Identify the top failure modes, including stale docs, policy ambiguity, unsupported answers, and permission leaks, with concrete mitigations.
