Diagnose and Improve a Flawed Model

Diagnose a flawed model, explain how to improve it, and discuss alternative methods that would perform better.

Asked in the Data Analytics and Intuition stage. This is a hybrid case study on machine learning and experimentation.

Address the following:

1. Clarify what information you need to diagnose the model.
2. Separate data, objective, modeling, evaluation, and serving problems.
3. Propose an improvement plan and explain how you would prioritize it.
4. Compare alternative modeling approaches and their tradeoffs.
5. Define offline and online experiments, monitoring, and rollback criteria.

Do not assume a particular product, dataset, model family, or deployment scale. State which assumptions and measurements you would request first.