Numerical Regression Test Harness | Dataford Interview Questions

How would you structure an automated test harness to catch numerical regressions in model training pipelines?

Implement this as a deterministic function that compares baseline and candidate metric dictionaries. A metric regresses when its directional change exceeds the larger of its absolute and relative tolerances; a missing candidate metric is also a regression. Return regression names in sorted order.

Constraints

1 <= len(baseline) <= 10^4
baseline, candidate, and rules use string metric names
Every baseline metric has a corresponding rule
Metric values and tolerances are finite numbers
direction is either "lower_is_better" or "higher_is_better"
candidate may omit baseline metrics, but extra candidate metrics are ignored

Function Signature

def find_numerical_regressions(baseline, candidate, rules):

Your question is Numerical Regression Test Harness. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

How would you structure an automated test harness to catch numerical regressions in model training pipelines?

Constraints

1 <= len(baseline) <= 10^4
baseline, candidate, and rules use string metric names
Every baseline metric has a corresponding rule
Metric values and tolerances are finite numbers
direction is either "lower_is_better" or "higher_is_better"
candidate may omit baseline metrics, but extra candidate metrics are ignored

Function Signature

def find_numerical_regressions(baseline, candidate, rules):

Your question is Numerical Regression Test Harness. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.