Debug Diverging Ad CTR Training

Business Context

OpenAI is training a click-through-rate model for in-product recommendation slots shown across ChatGPT surfaces. A recent training run on the latest week of data is diverging: training loss becomes nan after a few hundred steps, validation AUC drops below a historical baseline, and gradient norms spike by 100x relative to prior runs.

Dataset

You are given a tabular dataset built from impression logs for a binary classification task: predict whether a user clicks a recommended item within the session.

• Size: 2.4M impressions over 14 days, 48 features
• Target: Binary — clicked recommendation (1) vs not clicked (0)
• Class balance: 6.4% positive, 93.6% negative
• Missing data: 8% missing in user-history features for cold-start users; rare extreme outliers in count features

Success Criteria

A successful solution should identify the most likely causes of divergence, implement fixes, and restore stable training with validation performance at or above the previous benchmark: ROC-AUC = 0.78 and log loss = 0.21 on the held-out test set.

Constraints

• Nightly retraining must finish within 45 minutes on a single GPU or CPU-only fallback.
• The model should support probability calibration for ranking downstream.
• The debugging workflow must be reproducible and suitable for production incident response.

Deliverables

1. Propose a step-by-step debugging plan for the diverging run.
2. Build a stable training pipeline and explain each mitigation.
3. Compare a baseline linear model against a small neural network.
4. Report evaluation metrics and training diagnostics before and after fixes.
5. Recommend production monitors to catch future divergence early.