I calcaulate the RMSE values of n-times repeated 10-fold cross-validation (i.e. $n \times 10$ RMSE values) for two different models. Then, I perform a paired t-test on the RMSE values in order to determine if the mean RMSE of the models are different.

However, depening on n, I can always come up with a significant difference, which makes me question the validity of this approach.

One could argue that the magnitude of the estimate must be taken into account when selecting the model, but this is in my opinion a rather subjective choice and is not better than evaluating the model performance by eyeballing the RMSE distirbutions.

So, is the whole paired t-test procedure useful at all for model selection? Is there a scenario existing where such a procedure makes sense?

  • 4
    $\begingroup$ Consider: if I fit the linear regression y ~ x and then compare the residuals to the reduced regression y ~ 1, the average difference will be 0 every time even if x strongly predicts Y. So the null hypothesis of mean difference is true every time. The same problem happens with your approach. Cross-validation is about iteratively assessing predictive performance in models developed in one data set and validated in another. A mean difference is a bad measure of predictive accuracy. Squared mean difference on the other hand... $\endgroup$
    – AdamO
    Mar 9, 2018 at 15:55
  • $\begingroup$ @AdamO, I think he means squared error. and is concerned about statistical vs 'clinical' importance (can't remember the keyword). $\endgroup$
    – seanv507
    Mar 9, 2018 at 18:27
  • $\begingroup$ That's right. I rephrased the question to make it clearer. $\endgroup$
    – Funkwecker
    Mar 9, 2018 at 20:00
  • $\begingroup$ @AdamO Yes, but I am comparing RMSE values (not explictly stated in the question before) so the average difference of mean RMSE will not be 0 in your example. $\endgroup$
    – Funkwecker
    Mar 10, 2018 at 7:40
  • $\begingroup$ AdamO: should that be "mean squared difference on the other hand"? $\endgroup$ Mar 11, 2018 at 10:40

1 Answer 1


depening on n, I can always come up with a significant difference, which makes me question the validity of this approach.

There are two difficulties here.

  1. The first one is general: "significant" has a very specific meaning, and significance does not imply importance or relevance for the practical question at hand. So for significant differences you always need to make sure that they are also important/relevant for the task/question at hand.
    This applies to all hypothesis tests, not only to your model optimization.
    The recommended procedure is to specify beforehand what differences are consided relevant. Statistics cannot help with this, this information has to come from the application side. Once you know this, you can make sure that your test setting has enough power, i.e. that relevant differences would be considered significant as well.

  2. In the context of your model optimization there's a second catch. With your paired t-test you treat the $n$ RMSEs as independent of each other. But they are not (and were never meant to be - see below).

Proper setting for the paired $t$-test

The correct setting for the paired $t$-test are $n$ statistically independently measured RMSEs. This means you need to get $n$ test sets independent of each other, i.e. $n$ sets of $N$ test cases with all $n \cdot N$ cases independent of each other.

You then get predictions for all $n \cdot N$ test cases by each of your models, and then do the paired (because both models predicted exactly the same test cases) t-test.

Obviously, this procedure needs $(n + 1) \cdot N$ cases in total, so $n$ times more than you have at hand now. Or, you could train only on a tiny $\frac{1}{n+1}$ fraction of your cases. (And this is why noone does it.)

A second sensible setting for the paired t-test is doing it on a single run of the cross validation. $n$ is then the number of actual cases tested and the t-test is a valid procedure under the usual assumption of cross valiation: that all surrogate models are equivalent (i.e. training is stable) and the predictions can thus be pooled and considered as one test/validation experiment.

This stability assumption can easily be tested by:

repeated cross validation

So one basic assumption for all resampling validation schemes is that the generated surrogate models are equvialent (i.e. have the same predictive performance). This implies that if all surrogate models are tested with the same sample, their predictions should be equal.
And this is what you can check with repeated cross validation: you get $n$ predictions for each of your cases, and any difference between those predictions must be due to differences in the surrogate models. I.e. differences due to echanging a few of the training cases.

In other words, variance in predictions for the same case across the surrogate models tell you that your models are unstable. (The final RMSEs are not very good at spotting this, but if they vary, models are very unstable).

Now, there's a 2nd type of variance uncertainty on your RMSEs. And the crucial point where your t-test goes wrong is that the $n$ RMSEs you generaty by repeated cross validation do not reflect variance due to the limited number of test cases.
It is not, that this variance isn't there, but you cannot measure it this way, because all your RMSEs aggregate over the exact same test cases.

Take home message: there's no way to get around the fact that in total you have only $N$ independent cases in your data set.

Two more points we can conclude:

  • there's one scenario where your t-test approach would be OK: if the models are unstable but you have so many cases that the variance due to limited actual number of test cases can be neglected.
    However, this is practially irrelevant, as you'd first take care of stability to improve your model.

  • repeating cross validation helps the performance estimates only by averaging more surrogate models. I.e. only the uncertainty related to model instability is reduced, not the uncertainty due to the number of actual cases tested.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.