Significance test for comparing different 10-fold cross-validated Machine Learning Regressions

Question

Is there a recommended significance test for comparing different 10-fold cross validated regressions?

For instance, I want to compare the performance of LASSO against Random Forest for my dataset.

1. Both models are then evaluated using 10-fold cross validation.
2. Within each cross-validatoin, each one produces a different model,
3. Each model is compared against the test fold. Each test fold therefore has N / 10 comparisons.
4. Aggregating across the different models, I get N predictions, each of which I can then compare the ŷ against the y to see evaluate model accuracy.
5. If I square this differnce, This gives me (ŷ-y)^2, or the squared errors for each observation.
6. If i compare the mean of (ŷ-y)^2 for a modeling approach (LASSO vs RF), the i effectively get the MSE for each approach for my data
7. Normally, I would just select the model approach with the best MSE, and use that. Is there a way to test the significance between the two modeling approaches to confirm if one is producing a significantly lower MSE than the other?

My intuition is to run a paired-samples t-test on the (ŷ-y)^2 between the two methods. However, if that works as a test, won't running repeated cross-validations (and thus leading to an unlimited N mean that every comparison of models is infinitely good if you are as long as you run enough cross-validatoin repetitoins?

Also, what would I do if I wanted to compare a model tested on a 10-fold cross validated set against one that was tested on a 4-fold cross validated set (the 4-fold set is due to having 4 sites in a trial, and doing site-wise cross validation). In this case, the 10-fold cross validation can be repeated an unlimited times, since it is resampling. But the 4-fold is confine so the site-wise cross validation cannot be resample dan infinite number of times.

If I run a paired-samples t-test across a model run in the 10-fold vs the 4-fold to see accuracy, does it ruin the assumptions of the t-test?

Answer 1

Here's my take on the situation:

  won't running repeated cross-validations (and thus leading to an unlimited N mean that every comparison of models is infinitely good if you are as long as you run enough cross-validatoin repetitoins?

no, because (whatever test you decide to use), you need to adjust the degrees of freedom according to the actual number of independent test (in the CV sense) cases you have. Which does not increase with the repetitions, since no new cases enter the procedure.

In the CV situation there are several distinct sources of variance. In particular

• the (finite, limited) number of independent cases that are ever used as test case during the CV.
  This cannot increase beyond the number of cases that you encounter within a repetition of the CV. It may anyways be quite smaller than the number of data rows if you have dependence there e.g. because of repeated or nearly repeated measurements.

• Model instability: The variance in true performance between the CV surrogate models. Repetitions help to get a better assessment of this.

Now, there are two different scenarios for which such model comparisons are frequently used:

1. A model is built from the data set at hand for application use and its performance for production use is to estimated, e.g. in order so choose the best between a bunch of such models.
2. Model training algorithms are to be compared for a particular application domain. The crucial difference is that here we want to estimate the performance of the training algorithm for similar, but new data sets. So the training data set at hand is only one example/instance/sample.

For cross validation, those two scenarios make different assumptions:

Scenario 1 assumes that all surrogate models are equivalent to the production use model that is trained on the whole data set at hand. This assumption is often not met: the surrogate models are a bit worse due to having fewer training cases. And the surrogate models may not even be equivalent to each other. We can measure/estimate the variance between the performance of the surrogate models, though via the cross validation. We may that variance as approximation for the additional uncertainty wrt. the performance of the model derived from the whole data set. Note that the selection decision would not change with a constant bias across the considered algorithms.

Scenario 2 in contrast makes a far more dangerous assumption by taking re-samples that cannot differ by more than 2/k from each other as approximations of entirely new samples. Variance observed in the surrogate models is therefore a rather bad approximation for the actual variance needed to judge significance.

---

At the moment, I cannot provide you with a suitable significance test that accounts for these different sources of variance.

However, there are two situations where we can draw conclusions:

• If you are in situation 1, and the surrogate models are stable, i.e. variance in surrogate model performance << variance due to finite test sample size, you may neglect the former and proceed with a pairwise test using only the latter variance.
  A limitation is that the pessimistic bias of the cross validation may not be constant across the different training algorithms - and this is not taken into account.
  (If the situation is the opposite, the conclusion would be that you need to improve = stabilize the training rather than moving on to production use)

• If performance does not differ significantly under scenario 1, it cannot possibly be significant under scenario 2, neither, since scenario 2 is subject to additional variance uncertainty.