Thread-Safe Inference Caching | Dataford Interview Questions

How would you implement a thread-safe caching mechanism for pre-computed model inference results? Implement a bounded LRU cache with TTL-free entries and single-flight computation: concurrent requests for the same missing key must share one model computation. The function receives requests grouped into execution rounds, where requests within a round run concurrently and rounds execute sequentially. Return results in input order and the number of model computations per key.

Contract

requests is a list of {key, result, round} objects, and capacity is a positive integer. Implement run_inference_requests(requests, capacity) returning {results: list, compute_counts: dict}.

Constraints

0 <= len(requests) <= 10^4
1 <= capacity <= 10^4
Each request contains a string key, a JSON-compatible result, and a nonnegative integer round
Requests with the same key represent the same inference result
Model computations succeed

Function Signature

def run_inference_requests(requests, capacity):

Your question is Thread-Safe Inference Caching. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Contract

requests is a list of {key, result, round} objects, and capacity is a positive integer. Implement run_inference_requests(requests, capacity) returning {results: list, compute_counts: dict}.

Constraints

0 <= len(requests) <= 10^4
1 <= capacity <= 10^4
Each request contains a string key, a JSON-compatible result, and a nonnegative integer round
Requests with the same key represent the same inference result
Model computations succeed

Function Signature

def run_inference_requests(requests, capacity):

Your question is Thread-Safe Inference Caching. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.