Your question is Evaluating LLM Performance in Production. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
How would you evaluate the performance of a language model in production at New York Life? Explain how you would define evaluation datasets, select task-specific and operational metrics, and combine automated and human assessment. Address production monitoring, safety, privacy, distribution shift, latency, cost, and how evaluation results would guide model improvement. Distinguish offline validation from ongoing online evaluation and describe how you would handle cases where reliable labels are delayed or unavailable.