Explain Fine-Tuning vs RAG

Context

BrightDesk is building an internal AI assistant for sales engineers and product managers. One common use case is answering stakeholder questions like: "What's the difference between fine-tuning and RAG, and when should we use each?" in plain business language.

Constraints

• p95 latency: 1,500ms
• Cost ceiling: $3,000/month at 20,000 requests/month
• Hallucination rate: <2% on a 150-question golden set
• Answers must be understandable to non-technical stakeholders and avoid unnecessary jargon
• The assistant must not invent company capabilities, customer examples, or ROI claims
• Prompt injection and unsupported claims are considered real production risks

Available Resources

• A curated internal knowledge base with 40 short documents: AI glossary, architecture patterns, pricing notes, case studies, and approved messaging
• 200 historical stakeholder questions with human-written answers
• Access to a hosted LLM, embedding model, and vector search index
• PM and solutions engineering reviewers who can label a small evaluation set

Task

1. Propose whether you would solve this primarily with prompt design, RAG, fine-tuning, or a combination, and justify the choice for this use case.
2. Design an evaluation plan first: define offline and online metrics for clarity, factuality, hallucination, and stakeholder usefulness.
3. Write a system prompt that explains fine-tuning vs RAG in plain English, includes when to use each, and refuses unsupported business claims.
4. Describe the serving architecture, including whether retrieval is needed, how you would ground answers, and how you would keep latency and cost within budget.
5. Identify likely failure modes such as jargon-heavy answers, hallucinated examples, and prompt injection, and explain mitigations.