I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-fold cross-validation and perform a significance test.

My question is, which significance test should I use in the RMSE case?

  • $\begingroup$ What is your null hypothesis? $\endgroup$
    – Dave
    Sep 8 '20 at 18:11
  • $\begingroup$ @Dave That there is no statistical significance between the RMSE of the baseline model and each of the other models RMSE values $\endgroup$
    – MarG
    Sep 8 '20 at 18:15
  • $\begingroup$ So something like ANOVA but first RMSEs instead of means? (As a heads up, “statistically significant” is never part of a null hypothesis.) $\endgroup$
    – Dave
    Sep 8 '20 at 18:21
  • $\begingroup$ @Dave I was thinking that I would need to use a Wilcoxon sign test? $\endgroup$
    – MarG
    Sep 8 '20 at 19:01
  • $\begingroup$ With one observation per model? $\endgroup$
    – Dave
    Sep 8 '20 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.