Context
Cognition wants an internal assistant in Devin that helps engineering managers stay current on fast-moving GenAI developments: new model releases, prompting techniques, RAG patterns, agent frameworks, eval methods, and safety practices. The feature should produce a short weekly brief grounded in approved sources rather than generic summaries.
Constraints
- p95 latency per on-demand query: < 2,500ms
- Weekly digest generation cost ceiling: < $3,000/month for 500 managers
- Hallucination ceiling on cited claims: < 2% on a labeled eval set
- Prompt injection success rate from external content: 0 tolerated in production path
- Answers must distinguish between confirmed facts, vendor claims, and opinion/speculation
Available Resources
- Approved source set: Anthropic/OpenAI/Google model release notes, major research blogs, arXiv abstracts, selected GitHub repos, internal Cognition engineering notes
- 6 months of historical AI-news links with human-written summaries
- One frontier model for synthesis and one cheaper model for classification/filtering
- Existing search infrastructure with keyword + vector retrieval
- 20 internal users available to label a golden set of 150 questions and 50 weekly digest examples
Task
- Design an evaluation-first approach for an LLM system that answers: "How do you stay current on the latest trends in AI?" in a practical, reproducible way for Cognition managers using Devin.
- Write a system prompt that produces grounded trend summaries, cites sources, flags uncertainty, and refuses unsupported claims or attempts to follow instructions embedded in retrieved content.
- Propose the architecture for source ingestion, retrieval, ranking, synthesis, and digest generation, including how you would separate high-confidence factual updates from hype.
- Define offline and online metrics for relevance, freshness, citation faithfulness, hallucination rate, and usefulness to engineering managers.
- Estimate cost/latency and identify the main failure modes, especially stale information, source-quality drift, hallucinated trends, and prompt injection from web content.
