Business Context
Cisco’s Technical Assistance Center (TAC) supports enterprise customers running large networks (campus, data center, SD-WAN). TAC handles ~120,000 support cases/month globally, and customers increasingly expect self-service answers inside Cisco’s support portal. The goal is to build a chatbot that can answer technical troubleshooting questions using Cisco internal documentation (runbooks, bug notes, release notes, configuration guides, KB articles) and reduce time-to-resolution while avoiding unsafe or hallucinated guidance.
A wrong answer can cause customer downtime (SLA penalties, churn) or security exposure (misconfigured ACLs, weak crypto). The chatbot must therefore be grounded in internal sources, provide verifiable citations, and gracefully escalate when confidence is low.
Data Characteristics
- Corpus size: ~8M documents (PDF/HTML/Markdown), ~6 TB raw; updated daily.
- Document types: IOS-XE/IOS-XR/NX-OS configuration guides, TAC playbooks, field notices, bug scrub notes, command references.
- Text length: 200–30,000 tokens; many docs include tables, CLI blocks, syslog snippets.
- Domain vocabulary: interface names (Te1/0/48), protocols (OSPF/BGP/HSRP), feature flags, bug IDs (CSCvx12345), CVEs.
- Queries: short (1–2 sentences) to long (multi-paragraph) with pasted logs and configs.
- Languages: English first; 10–15% queries include non-English fragments.
Success Criteria
- Answer usefulness: ≥70% “resolved without human agent” for eligible intents (how-to, known issue, config guidance).
- Grounding: ≥95% of answers include at least one correct citation; hallucination rate <2% in offline eval.
- Latency: p95 < 2.0s end-to-end for typical queries (excluding long file uploads).
- Safety: Must refuse/redirect for destructive actions without confirmation (e.g., reload, erase, key rotation) and for out-of-scope requests.
Constraints
- Data governance: Internal docs and customer configs/logs must remain in Cisco-controlled VPC; no external API calls.
- Access control: Some docs are restricted by product line/customer contract; retrieval must enforce ACLs.
- Observability: Store prompts/answers with redaction; enable audit trails for citations.
- Deployment: Single-region active with multi-region failover; GPU budget limited (e.g., 2×A10 per shard).
Requirements (Deliverables)
- Propose an end-to-end RAG architecture (ingestion → indexing → retrieval → generation → safety/guardrails).
- Describe how you would chunk and embed Cisco docs (including tables/CLI blocks) and handle daily updates.
- Implement a baseline pipeline that:
- extracts entities (platform, OS, version, protocol, error codes, bug IDs),
- retrieves top-k passages with ACL filtering,
- generates an answer with citations.
- Define an evaluation plan: offline retrieval metrics + answer quality + hallucination checks + regression tests.
- Explain how you would handle:
- ambiguous questions (ask clarifying questions),
- conflicting sources (release notes vs runbook),
- long pasted logs/configs (summarize then retrieve).
