Readify, a content recommendation platform, recently implemented a new ranking model aimed at improving user engagement. The model's performance is evaluated using Normalized Discounted Cumulative Gain (NDCG) as the primary metric. After deploying the new model, the team observed a 5% increase in offline NDCG scores compared to the previous version.
| Metric | Previous Model | Current Model | Change |
|---|---|---|---|
| NDCG@10 | 0.75 | 0.79 | +5.33% |
| Average Reading Time | 4.2 min | 4.3 min | +2.38% |
| Click-Through Rate | 0.12 | 0.14 | +16.67% |
| User Retention Rate | 0.65 | 0.67 | +3.08% |
While the NDCG increase suggests improved ranking quality, the team needs to determine if this translates to a tangible increase in user reading time, which is crucial for engagement metrics. The business is concerned that offline metrics do not always reflect actual user behavior.