Optimizing Inference Speed | Dataford Interview Questions

How would you optimize a machine learning model for inference speed without sacrificing significant accuracy?

Implement optimize_inference(layers, accuracy_budget). Each layer contains optimization options represented by {"loss": int, "latency": int}, where option 0 is the baseline. Select exactly one option per layer, keep total accuracy loss at or below accuracy_budget, and return the option index selected for every layer. Minimize total latency.

layers is a list of option lists, and the budget and losses are nonnegative integers. A baseline option always exists, so a valid selection is guaranteed.

Example: layers = [[{"loss": 0, "latency": 100}, {"loss": 2, "latency": 70}], [{"loss": 0, "latency": 80}, {"loss": 3, "latency": 55}]], accuracy_budget = 4 returns [1, 0].

Constraints

1 <= len(layers) <= 200
1 <= len(layers[i])
0 <= accuracy_budget <= 1000
0 <= layers[i][j]['loss'] <= accuracy_budget
1 <= layers[i][j]['latency'] <= 10^9
Option 0 for every layer has loss 0

Function Signature

def optimize_inference(layers, accuracy_budget):

Your question is Optimizing Inference Speed. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem