Thread-Safe LLM Inference Cache | Dataford Interview Questions

Implement a thread-safe cache mechanism for an LLM inference endpoint.

Implement llm_cached_inference(requests, capacity, ttl). Each request contains prompt, model, parameters, time, and the response that a cache miss would produce. Return a dictionary containing responses in request order and the number of simulated inference calls. Requests with identical prompt, model, and parameters share a cache entry. Entries expire after ttl time units, and capacity eviction must follow LRU order. Protect all cache state with synchronization so concurrent access cannot corrupt it.

Input and Output

requests is a list of dictionaries. Return {"responses": [...], "inference_count": integer}.

Constraints

1 <= len(requests) <= 10^4
1 <= capacity <= 10^3
1 <= ttl
Each request contains prompt, model, parameters, time, and response
Request times are nondecreasing
Prompt and model are strings
Parameters and responses are JSON-compatible values

Function Signature

def llm_cached_inference(requests, capacity, ttl):

Your question is Thread-Safe LLM Inference Cache. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem