Welcome to your interview.
The question is on your right: Evaluate a Prompt Change. Take a moment with it first.
Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.
You maintain an LLM-powered feature and want to update the system prompt. You have a fixed set of 1,000 diverse user queries that represent real usage, and you need a reliable way to tell whether the new prompt is actually better or if it regresses performance on important cases.
How would you design a test suite to evaluate whether a prompt change improved or degraded model performance across 1,000 diverse user queries?