Is there a recommended significance test for comparing different 10-fold cross validated regressions?
For instance, I want to compare the performance of LASSO against Random Forest for my dataset.
- Both models are then evaluated using 10-fold cross validation.
- Within each cross-validatoin, each one produces a different model,
- Each model is compared against the test fold. Each test fold therefore has N / 10 comparisons.
- Aggregating across the different models, I get N predictions, each of which I can then compare the ŷ against the y to see evaluate model accuracy.
- If I square this differnce, This gives me (ŷ-y)^2, or the squared errors for each observation.
- If i compare the mean of (ŷ-y)^2 for a modeling approach (LASSO vs RF), the i effectively get the MSE for each approach for my data
- Normally, I would just select the model approach with the best MSE, and use that. Is there a way to test the significance between the two modeling approaches to confirm if one is producing a significantly lower MSE than the other?
My intuition is to run a paired-samples t-test on the (ŷ-y)^2 between the two methods. However, if that works as a test, won't running repeated cross-validations (and thus leading to an unlimited N mean that every comparison of models is infinitely good if you are as long as you run enough cross-validatoin repetitoins?
Also, what would I do if I wanted to compare a model tested on a 10-fold cross validated set against one that was tested on a 4-fold cross validated set (the 4-fold set is due to having 4 sites in a trial, and doing site-wise cross validation). In this case, the 10-fold cross validation can be repeated an unlimited times, since it is resampling. But the 4-fold is confine so the site-wise cross validation cannot be resample dan infinite number of times.
If I run a paired-samples t-test across a model run in the 10-fold vs the 4-fold to see accuracy, does it ruin the assumptions of the t-test?