Optimize Transformer Inference Latency | Dataford Interview Questions

How would you optimize the latency of an inference endpoint serving a large transformer model?

Implement optimize_latency(optimizations, memory_budget) to select compatible optimizations that maximize total latency reduction under a memory budget. Each optimization may require one prerequisite, and selecting an optimization requires selecting its entire prerequisite chain. Return the maximum achievable latency reduction.

Each optimization is a dictionary with integer fields memory, reduction, and requires, where requires is another index or -1. The prerequisite graph is a forest. Return an integer.

Constraints

1 <= len(optimizations) <= 200
0 <= memory_budget <= 10000
1 <= optimization.memory <= memory_budget or the optimization is unselectable
0 <= optimization.reduction <= 10^6
requires is -1 or a valid index different from the optimization index
The prerequisite graph is a forest with no cycles

Function Signature

def optimize_latency(optimizations, memory_budget):

Your question is Optimize Transformer Inference Latency. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

How would you optimize the latency of an inference endpoint serving a large transformer model?

Each optimization is a dictionary with integer fields memory, reduction, and requires, where requires is another index or -1. The prerequisite graph is a forest. Return an integer.

Constraints

1 <= len(optimizations) <= 200
0 <= memory_budget <= 10000
1 <= optimization.memory <= memory_budget or the optimization is unselectable
0 <= optimization.reduction <= 10^6
requires is -1 or a valid index different from the optimization index
The prerequisite graph is a forest with no cycles

Function Signature

def optimize_latency(optimizations, memory_budget):

Your question is Optimize Transformer Inference Latency. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.