Version Data and Models Reliably

Context

ShopLens runs a purchase propensity model that scores 4.2M users daily for email and in-app promotions. Over the last quarter, offline validation remained stable, but online campaign performance became inconsistent after multiple untracked data refreshes and model replacements.

The team currently stores model artifacts in object storage with ad hoc names and overwrites feature tables in place. Leadership wants a production-ready versioning strategy for both data and models that preserves reproducibility, supports rollback, and makes metric changes explainable.

Current Performance

The Problem

The team cannot determine whether the drop came from model changes, training data changes, feature definition drift, or threshold updates. There is no reliable mapping between a prediction in production and the exact training dataset, feature code, hyperparameters, or evaluation report used to create it.

Requirements

1. Diagnose what the metrics suggest about the likely failure modes.
2. Design a versioning approach for raw data, curated features, training datasets, model artifacts, and thresholds.
3. Specify what metadata must be logged to reproduce any model run and any production prediction batch.
4. Recommend validation checks before promotion and rollback criteria after deployment.
5. Explain how your design would support auditability, comparison across versions, and root-cause analysis.

Constraints

• Retraining happens weekly and must finish within 6 hours.
• Marketing can tolerate at most a 5% relative drop in conversion for one week.
• Raw event data cannot be modified after ingestion, but feature tables are currently mutable.