I have a series of Principal Components which I'm regressing on some data (it's a Blind Source Separation problem; I assume that my data is a linear combination of signals which I obtained via PCA), and I have to decide on the number of components for the model; I chose to do this via $K$-fold Cross-Validation.
I'm having problems because the number of components chosen via CV is highly sensitive to the choice of $K$, the number of folds of the CV procedure. I have $n=40$ datapoints and a maximum of 8 components to regress the data on. Here's an example of a $10$-fold Cross-Validation procedure:
In this case, the minimum CV error occurs using 8 components, but the value 1-SE away from this value is 3, so I might be tempted to use 3 components in this case. However, in this case the validation folds are too small, and that explains the high variance on the CV-error. If I instead use, e.g., $5$-fold CV, then the value 1-SE away from the minimum is 7 components; this number oscillates for each value of $K$ and so does the number of components where the global minimum is.
After looking at several CV plots for different $K$'s, I noted that there is always a local minimum at 4 components, followed by an increase in the CV-error on components $>4$, followed by a global minimum on components $7-8$. Considering that I'm using principal components, it seems reasonable to me to use this number as the 'optimal' number of components because this is the first minimum I encounter counting from the lowest to the highest number of components (considering that the number of components are ordered in terms of their eigenvalues). I find that this reasoning, however, is by no means 'clean' or an 'elegant' solution/explaination to the problem at hand; it just seems intuitive to me...do you have any advice on expanding this solution or proposing another one?