Testing significance of RMSE of models

I repeatedly trained two neural network models and calculated the RMSE for each run (split validation). Which statistical test is most useful in this case for testing if the difference of the models mean RMSE is significant? I think it is save to assume that the RMSE of the models is normally distributed.

Maybe welch test or two-sample t-test?

You can use the two sample t-test if the RMSE you get are normally distributed (you can check it before with the Shapiro-Wilk test for instance).

However, in your case, you want to know which model fits better your data. A usual way to do this is to use likelihood ratio statistics. Obviously, it requires to calculate properly the likelihoods of your two models, but if you assume Gaussian noises, it can be a quite easy task. Next, you make the ratio and compare with the quantile of a $\chi^2$ distribution.

• But isn't RMSE a measure of model fit? Commented Dec 17, 2015 at 18:36
• Yes, it is. You can express the log-likelihood in function of the RMSE under a Gaussian assumption, see here for instance: stats.stackexchange.com/questions/16508/…. In your problem, you have multiple runs, so you have to consider multivariate Gaussian noise when computing the log-likelihoods. Commented Dec 20, 2015 at 10:00
• Just another add-on question: Is using t-test correct anyways? The RMSE-distribution is bounded by zero on the left side and therefore does not follow a t-distribution, does it? Commented Jan 11, 2016 at 6:29
• Is using t-test correct anyways? The RMSE-distribution is bounded by zero on the left side and therefore does not follow a t-distribution, does it? Commented Dec 15, 2022 at 21:41