Is there a recommended significance test for comparing different 10-fold cross validated regressions?

For instance, I want to compare the performance of LASSO against Random Forest for my dataset.

  1. Both models are then evaluated using 10-fold cross validation.
  2. Within each cross-validatoin, each one produces a different model,
  3. Each model is compared against the test fold. Each test fold therefore has N / 10 comparisons.
  4. Aggregating across the different models, I get N predictions, each of which I can then compare the ŷ against the y to see evaluate model accuracy.
  5. If I square this differnce, This gives me (ŷ-y)^2, or the squared errors for each observation.
  6. If i compare the mean of (ŷ-y)^2 for a modeling approach (LASSO vs RF), the i effectively get the MSE for each approach for my data
  7. Normally, I would just select the model approach with the best MSE, and use that. Is there a way to test the significance between the two modeling approaches to confirm if one is producing a significantly lower MSE than the other?

My intuition is to run a paired-samples t-test on the (ŷ-y)^2 between the two methods. However, if that works as a test, won't running repeated cross-validations (and thus leading to an unlimited N mean that every comparison of models is infinitely good if you are as long as you run enough cross-validatoin repetitoins?

Also, what would I do if I wanted to compare a model tested on a 10-fold cross validated set against one that was tested on a 4-fold cross validated set (the 4-fold set is due to having 4 sites in a trial, and doing site-wise cross validation). In this case, the 10-fold cross validation can be repeated an unlimited times, since it is resampling. But the 4-fold is confine so the site-wise cross validation cannot be resample dan infinite number of times.

If I run a paired-samples t-test across a model run in the 10-fold vs the 4-fold to see accuracy, does it ruin the assumptions of the t-test?

  • $\begingroup$ This does not address the details of your question, but Diebold-Mariano test may be a relevant tool. $\endgroup$ Commented May 18, 2022 at 7:15
  • $\begingroup$ I'm avoiding the Deibold-Mariano specifically because the author of it says it's not for comparing models. "The Diebold–Mariano (DM) test was intended for comparing forecasts; it has been, and remains, useful in that regard. The DM test was not intended for comparing models. " tandfonline.com/doi/full/10.1080/07350015.2014.983236 $\endgroup$ Commented May 18, 2022 at 18:36
  • $\begingroup$ Can I just run a Wilcoxon T-test between the squared errors of the two different modelling approaches? My qualm with that is: if it's just a simple t-test, can't you endlessly inflate the sampel size by running more repeated cross validations, therefore always proving that they are significantly different? $\endgroup$ Commented May 18, 2022 at 18:39
  • $\begingroup$ Then if you follow the same paper, you should find some suggestions of how to do that in other ways that do not use the DM test. Regarding Wilcoxon t-test, I suspect the argument against the use of the DM test applies to the Wilcoxon t-test, too. $\endgroup$ Commented May 18, 2022 at 18:39
  • $\begingroup$ Thanks but those methods don't address the main issue with the repeated cross-validation MSE: Since i can repeatedly repeat cross validations with new samples each time, doesn't this allow endlessly inflating the sample? $\endgroup$ Commented May 18, 2022 at 18:51

1 Answer 1


Here's my take on the situation:

  won't running repeated cross-validations (and thus leading to an unlimited N mean that every comparison of models is infinitely good if you are as long as you run enough cross-validatoin repetitoins?

no, because (whatever test you decide to use), you need to adjust the degrees of freedom according to the actual number of independent test (in the CV sense) cases you have. Which does not increase with the repetitions, since no new cases enter the procedure.

In the CV situation there are several distinct sources of variance. In particular

  • the (finite, limited) number of independent cases that are ever used as test case during the CV.
    This cannot increase beyond the number of cases that you encounter within a repetition of the CV. It may anyways be quite smaller than the number of data rows if you have dependence there e.g. because of repeated or nearly repeated measurements.

  • Model instability: The variance in true performance between the CV surrogate models. Repetitions help to get a better assessment of this.

Now, there are two different scenarios for which such model comparisons are frequently used:

  1. A model is built from the data set at hand for application use and its performance for production use is to estimated, e.g. in order so choose the best between a bunch of such models.
  2. Model training algorithms are to be compared for a particular application domain. The crucial difference is that here we want to estimate the performance of the training algorithm for similar, but new data sets. So the training data set at hand is only one example/instance/sample.

For cross validation, those two scenarios make different assumptions:

Scenario 1 assumes that all surrogate models are equivalent to the production use model that is trained on the whole data set at hand. This assumption is often not met: the surrogate models are a bit worse due to having fewer training cases. And the surrogate models may not even be equivalent to each other. We can measure/estimate the variance between the performance of the surrogate models, though via the cross validation. We may that variance as approximation for the additional uncertainty wrt. the performance of the model derived from the whole data set. Note that the selection decision would not change with a constant bias across the considered algorithms.

Scenario 2 in contrast makes a far more dangerous assumption by taking re-samples that cannot differ by more than 2/k from each other as approximations of entirely new samples. Variance observed in the surrogate models is therefore a rather bad approximation for the actual variance needed to judge significance.

At the moment, I cannot provide you with a suitable significance test that accounts for these different sources of variance.

However, there are two situations where we can draw conclusions:

  • If you are in situation 1, and the surrogate models are stable, i.e. variance in surrogate model performance << variance due to finite test sample size, you may neglect the former and proceed with a pairwise test using only the latter variance.
    A limitation is that the pessimistic bias of the cross validation may not be constant across the different training algorithms - and this is not taken into account.
    (If the situation is the opposite, the conclusion would be that you need to improve = stabilize the training rather than moving on to production use)

  • If performance does not differ significantly under scenario 1, it cannot possibly be significant under scenario 2, neither, since scenario 2 is subject to additional variance uncertainty.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.