Diagnose Offline-Online Performance Gap

Context

GitLab deployed a gradient boosted classifier in GitLab Duo to predict whether an AI-generated code suggestion will be accepted by a developer. Offline validation looked strong, so the model replaced a heuristic ranker in production. After launch, online acceptance and user satisfaction dropped, despite offline metrics remaining high on the holdout set.

Current Performance

The Problem

You need to explain why the model generalizes well offline but underperforms in production, and determine whether the issue is caused by data drift, label mismatch, thresholding, calibration, serving bugs, or evaluation leakage.

Requirements

1. Interpret the metric pattern and identify the most likely failure modes.
2. Propose a step-by-step evaluation plan to isolate whether the gap comes from data, labels, model scoring, or serving.
3. Recommend how to validate your hypotheses using GitLab production logs and delayed labels.
4. Suggest concrete model or evaluation changes to recover online performance.
5. Explain what you would monitor going forward and which metrics should trigger rollback.

Constraints

• Acceptance labels arrive with a 7-day delay.
• The model serves under a 120 ms p95 latency budget.
• Product wants to avoid a large increase in low-quality GitLab Duo suggestions.