PayWave, a fintech company, launched a new fraud-screening rule and tested whether it increases the false-positive rate above the historical benchmark. A product manager sees a small p-value and says, “So there’s only a 2% chance the null hypothesis is true.”
Use the test results below to explain quantitatively why a p-value is not the probability that the null hypothesis is true. Show both the frequentist interpretation of the p-value and a separate Bayesian calculation for the probability that the null is true after seeing the data.
PayWave tested whether the false-positive rate is higher than the historical rate of 5.0%.
| Quantity | Value |
|---|---|
| Historical false-positive rate under null, | 0.050 |
| Number of reviewed transactions, | 400 |
| Observed false positives, | 30 |
| Observed sample rate, | 0.075 |
| Significance level, | 0.05 |
| Prior probability null is true, | 0.80 |
| Prior probability alternative is true, | 0.20 |
| Likelihood of seeing data this extreme under , | 0.25 |
Assume a one-sided hypothesis test: the team only cares whether the false-positive rate increased.