Nimbus Analytics launched a new stakeholder dashboard for regional sales directors who make weekly pricing and inventory decisions. The product team claims the redesign improved decision-making, but leadership wants evidence that the dashboard led to better decisions rather than simply higher usage.
A decision-support model was built to predict whether a stakeholder decision would be judged "high quality" 30 days later based on downstream business outcomes (margin, stockout rate, forecast error, and override reversals). The old dashboard and new dashboard were each used by comparable stakeholder groups for 8 weeks.
| Metric | Old Dashboard Cohort | New Dashboard Cohort | Change |
|---|---|---|---|
| Accuracy | 0.74 | 0.78 | +0.04 |
| Precision | 0.69 | 0.81 | +0.12 |
| Recall | 0.77 | 0.63 | -0.14 |
| F1 Score | 0.73 | 0.71 | -0.02 |
| AUC-ROC | 0.79 | 0.83 | +0.04 |
| Log Loss | 0.56 | 0.49 | -0.07 |
| Avg. weekly decisions per stakeholder | 18.4 | 22.7 | +23.4% |
| Decisions later rated high quality | 61% | 64% | +3 pts |
The new dashboard appears to improve ranking quality and precision, but recall fell sharply. Leadership needs to know whether the dashboard truly improved stakeholder decision-making or if it only helped on a narrower subset of easy decisions.