# Understanding the information lost during PCA on classification

I am testing a self-implemented PCA on a set of image data ($N \times d$, $N = 10000$ sample size, $d = 28 \times 28$ feature size). When I observed the classification accuracy, I found the accuracy increase somehow with the increased target reduced dimension $k$ and decresed from then on. See the image below:

The maximum accuracy corresponded to $PoV = 89\%$.

I also tried reducing the data to $k = d$, the accuracy was still much lower than the maximum showed in the graph.

Is these what is expected for PCA? Or my classifier is not working?

How can I interpret the "information lost" for different $k$ in PCA?

ps: I even tried the "princomp' built-in function, it yield similar result.

UPDATE:

Sorry for not stating clearly. For both training set itself and a separate test set as test set, I got similar result as in the graph.

• It would be nice if you define in your text what are $d$ and $k$. Also, why did you used tag bayesian? Describe you sequence - i.e. you first were doing PCA of features and then classifying objects based on the extracted PCs, or other way round? – ttnphns Oct 4 '13 at 8:00
• @ttnphns Sorry for the ambiguity. I used a bayesian classifier and maybe that is relevant to some people. As for the sequence, I classified based on different size of PCs, using the same data transformation for training set and test set. – Ray Oct 4 '13 at 8:11

If you are estimating classification accuracy by looking at a validation or test set separate from your training set, then this is precisely what you would expect to happen.

Initially, as you increase the number of principal components, you incorporate real information and your classification improves.

If you take too many principal components then you get overfitting, treating noise in the training set as useful information, and producing conclusions that not only fail to generalise to other data outside your training set but which are actually misleading.

• Thanks! Actually I am getting quite similar shapes for both testing on training set and test set. This indicates that it is not likely to be a overfitting problem, am I right? – Ray Oct 4 '13 at 7:59
• You are correct; this should not be happening on the training error. – Henry Oct 4 '13 at 8:01
• Can you give me a hint where things can go wrong for my problem? And if I am using training set to validate, when # of PCs $=$ original feature size, what is the expected accuracy? Thanks a million. – Ray Oct 4 '13 at 8:15

I assume that you present test data of a "tuning" test set, which is independent of the principle component analysis (PCA) and classifier training data (and independent of the test data for measuring the performance of the final classifier), such as the results of an inner-loop of double cross- or out-of-bootstrap-validation.

In that case, you probably observe a textbook-version of accuracy vs. complexity of the classifier (see e.g. The Elements of Statistical Learning, fig. 2.11 p. 38 in the version free for download:

• for too low complexity (few PCs), relevant information is excluded. The classifier is not as good as it could be.
• for too high complexity (many PCs), the PC scores include not only relevant information but also noise. The classifier overfits the training data (becomes unstable) and does not generalize well.

So far, this is my guess at what is going on. But you can actually measure whether my guess is right:

You can distinguish these two situations by comparing training error (goodness-of-fit) and independent test error: in the first case, you see a lack of fit and a low generalization performance. In the second case, you see excellent fit, but nevertheless low generalization performance.
You can also directly measure model stability: If you resample your training data, you also see large variations between models trained on slightly different data sets (resampling exchanges some cases against others): the models are unstable. Instead of directly comparing the models, you can also compare predictions of the different "surrogate" models (built on different resampled training sets) for the same test cases.

When you do PCA, you get components that are sorted with decreasing variance. Doing this as a pre-processing step before classification is lead by the assumption that the relevant differences between the classes are the largest variations in the data set, whereas the higher componentents carry only noise. In that scenario, if the right number of PCs is chosen, only noise is cut away while the information (= variance relevant for the classification) is preserved.

So the drop for higher numbers of PCs is not due to lost information but to too high classifier complexity. Your data set probably does not carry enough information to train a stable classifier with the hight complexity corresponding to the high number of PCs.

• Thanks for your analysis. If my described accuracy in the graph also happens for validation set $=$ training set, where can the problem be? – Ray Oct 4 '13 at 8:19