I saw a paper comparing two ML models based on a statistical test. They tested these two models using different datasets and then they compared their accuracy results using a statistical test to see if their is any statistical significant. I was thinking if i were the author, I would just calculate the mean of the accuracy results and compare the two models based on their means. So, when to use statistical test and when to use simpler approaches like comparing the two means?


1 Answer 1


Quoting Francis X. Diebold "Comparing predictive accuracy, twenty years later: A personal perspective on the use and abuse of Diebold–Mariano tests" (2015),

We’ve all seen hundreds of predictive horse races, with one or the other declared the “winner” (usually the new horse in the stable), but with no consideration given to the statistical significance of the victory. Such predictive comparisons are incomplete and hence unsatisfying. That is, in any particular realization, one or the other horse must emerge victorious, but one wants to know whether the victory is statistically significant. That is, one want to know whether a victory “in sample” was merely good luck, or truly indicative of a difference “in population.”"

(Emphasis is mine.)

So if you just calculated the mean of the accuracy results and compared the two models based on their means, you would not know how likely the apparent superiority of one model is due to sampling variation versus the genuine difference between the forecasting ability of the models. Statistical tests provide tools for assessing this.

So, when to use statistical test and when to use simpler approaches like comparing the two means?

If you want an answer with some certainty bounds (which I guess is always the case), you need a test. If you skip the test, you have no way of telling how trustworthy the result is, making it rather useless.

  • $\begingroup$ @nbro, The null hypothesis postulates equally good predictive performance in population, or in this case, for a given data generating process. The setup: we have a process that is generating the data; we compare two models on a sample generated by the process; we make inference about forecasts on an infinitely large sample from the same process. $\endgroup$ May 6, 2019 at 17:45
  • $\begingroup$ @nbro, OK. Does that mean you have answered your question yourself, or is there anything left unanswered? $\endgroup$ May 6, 2019 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.