cross-validation analysis not diagnostic

I'm using k-fold cross-validation analysis for model selection, however, it does not appear to favor any particular model. There are several variants of the models and two of them are nested within (i.e., more restrictive version of) the third. I've tried using a different numbers of fold (2, 5, and 10) and multiple iterations (up to 100) with random splits of the data, but this does not appear to make a difference. I'm using the mean squared error of prediction (MSEP) to compare the models as well as the standard deviation of the squared error of prediction across iterations to get a sense of how noisy the MSEP is. So for, instance, the MSEP for model A and B may be .045 and .054, respectively, but they are both within one SD of each other. This makes me think that these differences are just random.

Does anyone have a sense of how to interpret this? If a more flexible model does as well as a more parsimonious model in cross-validation, does this mean that the simpler model should be favored? Or is possible that the cross-validation analyses are not diagnostic for these data? The number of observations is in the thousands and the data are used to construct proportions within different categories.

• The simpler model is (almost) always preferred. What is your sample size? Can you show us some plots? It may be, as you mentioned, that your model isn't really learning anything and this is all random. Commented Jan 7, 2019 at 9:51
• Thanks for the response. What kind of graphs would help? Commented Jan 7, 2019 at 22:48
• Show us how model prediction estimates change with various CV parameters, so for ex. x-axis is k=2,5,10,... and y-axis is MSEP. Averages and standard deviations. Commented Jan 8, 2019 at 9:06
• The link is pasted below. The number of observations is in the thousands. Model A is nested in Models B and C, and Model B is nested in Model C. <savepice.ru/full/2019/1/10/…> Commented Jan 9, 2019 at 21:27
• There is almost surely no difference between the 3 models, they are very similar, but what is weird is that your MSEP increases when increasing fold number, which should not happen, the opposite should be true, so you may just be introducing noise by adding more variables. Commented Jan 10, 2019 at 7:30

If you are comparing/selecting from 4 models that were all set up according to your knowledge of the application at hand as good and sensible candidates for a good model, there's really nothing that will guarantee different performance. After all, there can be various models that reach Bayes error for a given application (which doesn't mean you got there, there are more possibilities that can lead to equal or very similar predictive performance of models of different type).

I'd suggest looking a bit more into that relation of $$MSE_{CV}$$ (at least in my field, $$MSE_P$$ is reserved for test set predictions) vs $$k$$.
The fewer actual training cases available for surrogate model training may lead to worse models for small $$k$$, which may be more unstable and/or worse on average. However, you observe the opposite.