Context

A Databricks team runs a production RAG assistant for internal support using the Databricks Agent Framework, Databricks Vector Search, DBRX through Databricks Foundation Model APIs, and Databricks Model Serving. Source documents are stored in Delta Lake and governed by Unity Catalog. Over the last 3 weeks, user satisfaction has dropped even though generation latency and model availability are stable.

Offline evaluation with MLflow Agent Evaluation (mlflow.evaluate) suggests the retrieval layer may be degrading: the assistant increasingly answers with plausible but weakly supported content. The team wants a monitoring plan that detects retrieval failures before they materially impact users.

Current Performance

The Problem

You need to design an evaluation and monitoring approach that can distinguish retrieval failures from generation failures, quantify impact, and trigger investigation when the vector search index or retrieval pipeline starts drifting.

Requirements

1. Identify which metrics are leading indicators of retrieval failure versus lagging user-impact metrics.
2. Propose an online and offline monitoring design using MLflow, Delta Lake, and Databricks-native evaluation tooling.
3. Define alert thresholds and how you would reduce false alarms.
4. Explain how you would segment errors by query type, document source, and index freshness.
5. Recommend concrete remediation steps if retrieval quality continues to decline.

Constraints

1. The assistant serves 1.8M queries/day.
2. Human labeling is limited to 1,000 sampled queries/week.
3. Source documents update hourly via Apache Spark pipelines into Delta Lake.
4. The team cannot materially increase P95 latency or serving cost.