I have a series of Principal Components which I'm regressing on some data (it's a Blind Source Separation problem; I assume that my data is a linear combination of signals which I obtained via PCA), and I have to decide on the number of components for the model; I chose to do this via $K$-fold Cross-Validation.

The problem

I'm having problems because the number of components chosen via CV is highly sensitive to the choice of $K$, the number of folds of the CV procedure. I have $n=40$ datapoints and a maximum of 8 components to regress the data on. Here's an example of a $10$-fold Cross-Validation procedure:

enter image description here

In this case, the minimum CV error occurs using 8 components, but the value 1-SE away from this value is 3, so I might be tempted to use 3 components in this case. However, in this case the validation folds are too small, and that explains the high variance on the CV-error. If I instead use, e.g., $5$-fold CV, then the value 1-SE away from the minimum is 7 components; this number oscillates for each value of $K$ and so does the number of components where the global minimum is.

My approach

After looking at several CV plots for different $K$'s, I noted that there is always a local minimum at 4 components, followed by an increase in the CV-error on components $>4$, followed by a global minimum on components $7-8$. Considering that I'm using principal components, it seems reasonable to me to use this number as the 'optimal' number of components because this is the first minimum I encounter counting from the lowest to the highest number of components (considering that the number of components are ordered in terms of their eigenvalues). I find that this reasoning, however, is by no means 'clean' or an 'elegant' solution/explaination to the problem at hand; it just seems intuitive to me...do you have any advice on expanding this solution or proposing another one?

  • $\begingroup$ In case you are still interested in that: can you describe how exactly you did the cross-validation? How was the CV error computed? One has to be careful to do it correctly for PCA, and I suspect you might have done it wrong. $\endgroup$ – amoeba says Reinstate Monica Sep 12 '14 at 10:17
  • $\begingroup$ In addition to my previous question: are you doing CV to find an optimal number of PCA components (independent of what you are doing with these components afterwards)? Or are you using your components for something else, and the CV includes this other stuff as well? $\endgroup$ – amoeba says Reinstate Monica Sep 15 '14 at 16:12
  • Does your data have a structure that allows interpretation of the loadings? That is, can you check the loadings and see whether they can possibly carry information?

  • Where do the error bars come from? Different folds or different repetitions/iterations of the complete cross validation?

  • You observe that the CV results oscillate depending on the number of folds. How much is this "oscillation" compared to the variation you observe for iterations of CV with the same K (that is, new random assignments of the cases to the K folds)?

  • The concern with 4 components is that it may already be one component too much, and this could lead to overfitting/unstable models and thus to worse predictions. How does the training (resubstitution) error evolve? Is it lower than the cross validation results? Where does it start to show differences?

  • (Remember that you need a separate test set or independent outer cross validation to measure the predictive ability of the final model you build)

  • $\begingroup$ Hi, answering your questions; (1) It's really hard to tell. The structure of the loadings is complex enough to not be anything like I would expect them to be. (2) The error bars come from different folds. I take the model with $p$ components and fit it to the $k$-th fold, from $k=1$ to $k=K$. Each of these fits gives me a mean cv-error, which I then average over all the folds; from there I obtain the standard errors. $\endgroup$ – Néstor Feb 20 '13 at 21:28
  • $\begingroup$ (3) It's pretty stable, although for smaller $K$ it tends to select 7 to 8 components, which is way too many components for my application. (4) I didn't get this part, what do you mean by 'how does the training (resubstitution) error EVOLVE'? $\endgroup$ – Néstor Feb 20 '13 at 21:33
  • $\begingroup$ @Néstor: I think you should do several runs of the randomly split CV for 1 to 8 components and plot the differences between those runs (disregarding the recommendations of the CV). As each run of the CV tests each case exactly once, the variation between the runs tells you whether the models are unstable, which is another indicator of too many components. $\endgroup$ – cbeleites supports Monica Feb 20 '13 at 21:34
  • $\begingroup$ @Néstor: pretty stable, but differences in the error bars depending on $k$ is weird (unless there is really a systematic dependency on k - which would be a rather alarming situation [leave one out is known to show such behaviour in certain cases, though]). With (4) I mean, plot resubstitution error in the same graph. Is it much different, does it get more different (more overoptimistic) with more components? $\endgroup$ – cbeleites supports Monica Feb 20 '13 at 21:38
  • $\begingroup$ (you do take into account for the standard error that the total no. of cases is the same independent of K, i.e. that you calculate standard error for a mean of means, right?) $\endgroup$ – cbeleites supports Monica Feb 20 '13 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.