Your question is Evaluating an Deployed LLM. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
How would you evaluate the performance of a large language model (LLM) you deployed at Delta Air Lines?
Explain how you would define success criteria, construct representative evaluation data, select appropriate offline and online metrics, and monitor the model after deployment. Address quality, safety, latency, cost, reliability, and human review. Distinguish deterministic metrics from human and production-based evaluation, and explain how evaluation results would guide model improvements.
