StreamLine runs a personalized home feed that ranks posts, videos, and articles for each user session. A new recommendation model increased click-through rate after launch, but product leaders now see weaker downstream engagement and more user complaints about repetitive content.
| Metric | Previous Model | New Model | Change |
|---|---|---|---|
| CTR@10 | 6.8% | 8.1% | +1.3 pp |
| Save rate | 2.4% | 1.9% | -0.5 pp |
| Share rate | 1.1% | 0.8% | -0.3 pp |
| 7-day retention | 41.2% | 39.6% | -1.6 pp |
| Avg session time | 18.4 min | 17.1 min | -1.3 min |
| Hide / Not interested rate | 3.2% | 4.7% | +1.5 pp |
| Creator diversity@20 | 12.6 | 8.9 | -29.4% |
| NDCG@10 (offline) | 0.412 | 0.438 | +0.026 |
| Calibration error | 0.031 | 0.089 | +0.058 |
The ranking model appears better on top-line click metrics, but worse on satisfaction, diversity, and retention. You need to determine whether the model is genuinely better, over-optimized for clicks, or poorly calibrated in production.