cross-validation analysis not diagnostic

Question

I'm using k-fold cross-validation analysis for model selection, however, it does not appear to favor any particular model. There are several variants of the models and two of them are nested within (i.e., more restrictive version of) the third. I've tried using a different numbers of fold (2, 5, and 10) and multiple iterations (up to 100) with random splits of the data, but this does not appear to make a difference. I'm using the mean squared error of prediction (MSEP) to compare the models as well as the standard deviation of the squared error of prediction across iterations to get a sense of how noisy the MSEP is. So for, instance, the MSEP for model A and B may be .045 and .054, respectively, but they are both within one SD of each other. This makes me think that these differences are just random.

Does anyone have a sense of how to interpret this? If a more flexible model does as well as a more parsimonious model in cross-validation, does this mean that the simpler model should be favored? Or is possible that the cross-validation analyses are not diagnostic for these data? The number of observations is in the thousands and the data are used to construct proportions within different categories.

Answer 1

If you are comparing/selecting from 4 models that were all set up according to your knowledge of the application at hand as good and sensible candidates for a good model, there's really nothing that will guarantee different performance. After all, there can be various models that reach Bayes error for a given application (which doesn't mean you got there, there are more possibilities that can lead to equal or very similar predictive performance of models of different type).

However, what made me write this late answer is:

I'd suggest looking a bit more into that relation of $MSE_{CV}$ (at least in my field, $MSE_P$ is reserved for test set predictions) vs $k$.

• CV error really should not increase systematically with k,
• and standard deviation across iterations (as opposed to std across folds) measures instability of your models (or more precisely, their predictions).

The fewer actual training cases available for surrogate model training may lead to worse models for small $k$, which may be more unstable and/or worse on average. However, you observe the opposite.

This could be a symptom of some programming mistake, and programming mistakes when calculating model perforance can cause the whole verification/selection/optimization to go wrong.
(I'm speaking as someone who once retractred an already submitted manuscript after finding a tiny single character typo that created a data leak and caused severe optimistic bias in cross validation results).