3
$\begingroup$

I repeatedly trained two neural network models and calculated the RMSE for each run (split validation). Which statistical test is most useful in this case for testing if the difference of the models mean RMSE is significant? I think it is save to assume that the RMSE of the models is normally distributed.

Maybe welch test or two-sample t-test?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

You can use the two sample t-test if the RMSE you get are normally distributed (you can check it before with the Shapiro-Wilk test for instance).

However, in your case, you want to know which model fits better your data. A usual way to do this is to use likelihood ratio statistics. Obviously, it requires to calculate properly the likelihoods of your two models, but if you assume Gaussian noises, it can be a quite easy task. Next, you make the ratio and compare with the quantile of a $\chi^2$ distribution.

Improve this answer
$\endgroup$
4
  • $\begingroup$ But isn't RMSE a measure of model fit? $\endgroup$
    – Funkwecker
    Dec 17, 2015 at 18:36
  • 1
    $\begingroup$ Yes, it is. You can express the log-likelihood in function of the RMSE under a Gaussian assumption, see here for instance: stats.stackexchange.com/questions/16508/…. In your problem, you have multiple runs, so you have to consider multivariate Gaussian noise when computing the log-likelihoods. $\endgroup$
    – Jacky1
    Dec 20, 2015 at 10:00
  • 1
    $\begingroup$ Just another add-on question: Is using t-test correct anyways? The RMSE-distribution is bounded by zero on the left side and therefore does not follow a t-distribution, does it? $\endgroup$
    – Funkwecker
    Jan 11, 2016 at 6:29
  • $\begingroup$ Is using t-test correct anyways? The RMSE-distribution is bounded by zero on the left side and therefore does not follow a t-distribution, does it? $\endgroup$
    – There
    Dec 15, 2022 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.