Assess Offline NDCG Impact on User Reading Time

Context

Readify, a content recommendation platform, recently implemented a new ranking model aimed at improving user engagement. The model's performance is evaluated using Normalized Discounted Cumulative Gain (NDCG) as the primary metric. After deploying the new model, the team observed a 5% increase in offline NDCG scores compared to the previous version.

Current Performance

The Problem

While the NDCG increase suggests improved ranking quality, the team needs to determine if this translates to a tangible increase in user reading time, which is crucial for engagement metrics. The business is concerned that offline metrics do not always reflect actual user behavior.

Requirements

1. Analyze the correlation between NDCG improvements and changes in average reading time.
2. Propose a method to validate whether the offline NDCG increase leads to real-world engagement.
3. Identify any potential biases in the evaluation metrics used.
4. Recommend further experiments or analyses to confirm the model's effectiveness.

Constraints

• User engagement must be improved without negatively impacting system performance or increasing operational costs.