Tuning LLM Outputs | Dataford Interview Questions

An LLM is producing wrong/inaccurate short answers: examine the prompt and tweak its hyperparameters (temperature, number of tokens, etc.) to fix it.

Asked in the onsite test stage.

Implement choose_llm_hyperparameters(prompt, trials). prompt is a string. trials is a list of objects containing prompt, temperature, max_tokens, response, and expected. Consider only trials whose prompt exactly matches the target. Normalize response and expected by trimming, collapsing whitespace, and ignoring case. Return the configuration with the highest exact-match accuracy. Break ties by lower temperature, then lower max_tokens. Return None if no matching trials exist.

Example: matching configurations with accuracies 1.0 and 0.5 return the first configuration.

Constraints

1 <= len(trials) <= 1000
Each trial contains the five required keys
temperature is a nonnegative number
max_tokens is a positive integer
At least one matching trial may be absent
Responses and expected answers are strings

Function Signature

def choose_llm_hyperparameters(prompt, trials):

Your question is Tuning LLM Outputs. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem