When to use statistical tests to compare two machine learning models

Question

I saw a paper comparing two ML models based on a statistical test. They tested these two models using different datasets and then they compared their accuracy results using a statistical test to see if their is any statistical significant. I was thinking if i were the author, I would just calculate the mean of the accuracy results and compare the two models based on their means. So, when to use statistical test and when to use simpler approaches like comparing the two means?

Answer 1

Quoting Francis X. Diebold "Comparing predictive accuracy, twenty years later: A personal perspective on the use and abuse of Diebold–Mariano tests" (2015),

We’ve all seen hundreds of predictive horse races, with one or the other declared the “winner” (usually the new horse in the stable), but with no consideration given to the statistical significance of the victory. Such predictive comparisons are incomplete and hence unsatisfying. That is, in any particular realization, one or the other horse must emerge victorious, but one wants to know whether the victory is statistically significant. That is, one want to know whether a victory “in sample” was merely good luck, or truly indicative of a difference “in population.”"

(Emphasis is mine.)

So if you just calculated the mean of the accuracy results and compared the two models based on their means, you would not know how likely the apparent superiority of one model is due to sampling variation versus the genuine difference between the forecasting ability of the models. Statistical tests provide tools for assessing this.

So, when to use statistical test and when to use simpler approaches like comparing the two means?

If you want an answer with some certainty bounds (which I guess is always the case), you need a test. If you skip the test, you have no way of telling how trustworthy the result is, making it rather useless.