Selecting number of input variables in a PCA-LDA model

Question

Let's say that you have a dataset with a huge number of variables, so that a linear discriminant analysis on the original data may not be a good idea. If you first use PCA a dimensionality reduction method, you can use a certain number of PCA axes as the input variables for your LDA model (I think it's called PCA-LDA). But then, is there any usual criterion to know (and justify) what would be the optimal number of principal components to keep in your model?

As an example, I did that:

1. Perform PCA on original data.
2. For $p$ from 2 to [some reasonable number], use the $p$ first axes of the PCA as the input variables in an LDA model, and compute the classification error in k-fold cross-validation for each model size. (Side note: maybe I should rather recompute PCA at each iteration because it's considered to be part of the model?)

I got the following results:

Starting from here, it seems that maximal accuracy in cross-validation is reached for 20 principal components. But it also seems that retaining only 5 principal axes leads to a similar classification accuracy, with a very smaller model. In such a case, have we some "standard" measure to make a compromise between model size and model accuracy, that would allow to know how many components I should keep? (I mean, something like adjusted R-squared in regression, or Breiman's "1-SE rule" for CART algorithm, etc.)

Thanks!

Answer 1

maybe I should rather recompute PCA at each iteration because it's considered to be part of the model

YES! You definitively need to do this (at least unless you have a vast amount of experience with this type of data, and have repeatedly shown in the past that it doesn't matter - which would typically occur only in n >> p situations where the PCA isn't really needed)

PCA is likely to create a very strong dependency between cases/rows on moderate to small sample sizes (compared to dimensionality). And that is likely the case: if it weren't you could usually do the LDA without PCA first.

I usually have wide (n < p) data, and small sample size situations (though something close to repeated measurements), and I've seen the error rate underestimated by about an order of magnitude when the cross validation was not including the PCA.

---

There are various heuristics to optimize model complexity here. The key is IMHO to get some guesstimate of the uncertainty of the measured performance and account for this in the decision.

• one fairly widely used one is the one-standard-deviation rule as discussed e.g. in the Elements of Statistical Learning.

• In your case, I'd say the absolute number of cases is likely the bottleneck, which I guess to be roughly 45 from the steps in the data.

A back of the envelope calculation for the 5 PC model gives a 95 % confidence interval of very roughly 85 - 100 % based on 43 correct out of 45 tested cases. Which immediately tells you that you need not consider anything above 5 PCs here.

(You could set up more precise tests, e.g. McNemar's test to check whether the improvement 4 -> 5 PCs is significant; key would be to keep the number of tests low and penalize multiple testing)

Another component to the uncertainty is (in)stability, which you may want to consider more or less independently of the sample size consideration. This is possible from the predictions in repeated k-fold cross valiation, or from model coefficients of the "whole" PCA-LDA model after aligning the models in a way that keeps them equivalent (prediction of PCA-LDA models is invariant to flipping and rotation in LD-score space).

I work with spectroscopic data, where I can often see whether the coefficient vectors (for the full PCA-LDA projection) get noisy, but this is a specialty of the particular kinds of spectroscopic data I work with.

Two papers where we used the 1st and 2nd approach:

• Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations Anal Bioanal Chem, 2008, 390, 1261-1271.
  DOI: 10.1007/s00216-007-1818-6
• C. Beleites, K. Geiger, M. Kirsch, S. B. Sobottka, G. Schackert and R. Salzer: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal. Bioanal. Chem., 400 (2011), 2801 - 2816.
  manuscript without paywall

---

(I do have a sneaking suspicion that due to the very close link between PCA and LDA, PCA-LDA may not be that much more stable than LDA in general. But I haven't properly thought about this yet.)