1
$\begingroup$

Let's say that you have a dataset with a huge number of variables, so that a linear discriminant analysis on the original data may not be a good idea. If you first use PCA a dimensionality reduction method, you can use a certain number of PCA axes as the input variables for your LDA model (I think it's called PCA-LDA). But then, is there any usual criterion to know (and justify) what would be the optimal number of principal components to keep in your model?

As an example, I did that:

  1. Perform PCA on original data.
  2. For $p$ from 2 to [some reasonable number], use the $p$ first axes of the PCA as the input variables in an LDA model, and compute the classification error in k-fold cross-validation for each model size. (Side note: maybe I should rather recompute PCA at each iteration because it's considered to be part of the model?)

I got the following results:

enter image description here

Starting from here, it seems that maximal accuracy in cross-validation is reached for 20 principal components. But it also seems that retaining only 5 principal axes leads to a similar classification accuracy, with a very smaller model. In such a case, have we some "standard" measure to make a compromise between model size and model accuracy, that would allow to know how many components I should keep? (I mean, something like adjusted R-squared in regression, or Breiman's "1-SE rule" for CART algorithm, etc.)

Thanks!

$\endgroup$
3
  • 1
    $\begingroup$ This depends also on how many observations you have. Your plot suggests that numbers are pretty discrete, there may be just one observation difference between the about 0.95 you get at 5 PCs and the about 0.96 (?) you get at 20 PCs. Chances are these are not "significantly" different (although formally assessing that is difficult because of nonstandard dependence between these rates), in which case I'd go for 5 for sure. Generally you've got to live with the fact that there are no "objective standard rules" for this kind of thing; whatever people recommend will be somewhat arbitrary. $\endgroup$ – Lewian Mar 19 at 11:58
  • $\begingroup$ Thanks for your comment! You're right, I have a small-to-moderate sample size. I'm comfortable with arbitrary rules (as long as they are reasonable) in this kind of situation; just wondered about a possible consensus here (and proper justification to give in a paper and/or to a reviewer). But that's okay for me! $\endgroup$ – Philopolis Mar 19 at 12:35
  • 1
    $\begingroup$ If the difference in misclassifications is just one or two, you'll probably get away with saying "there's no significant improvement" without running a formal test. Actually one could run a test for equality of proportions, but this is techically wrong because of dependence of events. $\endgroup$ – Lewian Mar 19 at 13:02
2
$\begingroup$

maybe I should rather recompute PCA at each iteration because it's considered to be part of the model

YES! You definitively need to do this (at least unless you have a vast amount of experience with this type of data, and have repeatedly shown in the past that it doesn't matter - which would typically occur only in n >> p situations where the PCA isn't really needed)

PCA is likely to create a very strong dependency between cases/rows on moderate to small sample sizes (compared to dimensionality). And that is likely the case: if it weren't you could usually do the LDA without PCA first.

I usually have wide (n < p) data, and small sample size situations (though something close to repeated measurements), and I've seen the error rate underestimated by about an order of magnitude when the cross validation was not including the PCA.


There are various heuristics to optimize model complexity here. The key is IMHO to get some guesstimate of the uncertainty of the measured performance and account for this in the decision.

  • one fairly widely used one is the one-standard-deviation rule as discussed e.g. in the Elements of Statistical Learning.

  • In your case, I'd say the absolute number of cases is likely the bottleneck, which I guess to be roughly 45 from the steps in the data.

A back of the envelope calculation for the 5 PC model gives a 95 % confidence interval of very roughly 85 - 100 % based on 43 correct out of 45 tested cases. Which immediately tells you that you need not consider anything above 5 PCs here.

(You could set up more precise tests, e.g. McNemar's test to check whether the improvement 4 -> 5 PCs is significant; key would be to keep the number of tests low and penalize multiple testing)

Another component to the uncertainty is (in)stability, which you may want to consider more or less independently of the sample size consideration. This is possible from the predictions in repeated k-fold cross valiation, or from model coefficients of the "whole" PCA-LDA model after aligning the models in a way that keeps them equivalent (prediction of PCA-LDA models is invariant to flipping and rotation in LD-score space).

I work with spectroscopic data, where I can often see whether the coefficient vectors (for the full PCA-LDA projection) get noisy, but this is a specialty of the particular kinds of spectroscopic data I work with.

Two papers where we used the 1st and 2nd approach:


(I do have a sneaking suspicion that due to the very close link between PCA and LDA, PCA-LDA may not be that much more stable than LDA in general. But I haven't properly thought about this yet.)

$\endgroup$
2
  • $\begingroup$ Many thanks for this answer and the references! I've all I need now. Just a last question about recomputing PCA at each iteration: if I finally choose out-of-bootstrap or iterated-k-fold as CV procedure, one way to do that would be to compute the PCA at each iteration using only the training subset as the active individuals, and the test subset as supplementary individuals in the PCA? $\endgroup$ – Philopolis Mar 21 at 9:39
  • 1
    $\begingroup$ @Philopolis Yes, that would be correct. $\endgroup$ – cbeleites unhappy with SX Mar 22 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.