I assume that you present test data of a "tuning" test set, which is independent of the principle component analysis (PCA) and classifier training data (and independent of the test data for measuring the performance of the final classifier), such as the results of an inner-loop of double cross- or out-of-bootstrap-validation.
In that case, you probably observe a textbook-version of accuracy vs. complexity of the classifier (see e.g. The Elements of Statistical Learning, fig. 2.11 p. 38 in the version free for download:
- for too low complexity (few PCs), relevant information is excluded. The classifier is not as good as it could be.
- for too high complexity (many PCs), the PC scores include not only relevant information but also noise. The classifier overfits the training data (becomes unstable) and does not generalize well.
So far, this is my guess at what is going on. But you can actually measure whether my guess is right:
You can distinguish these two situations by comparing training error (goodness-of-fit) and independent test error: in the first case, you see a lack of fit and a low generalization performance. In the second case, you see excellent fit, but nevertheless low generalization performance.
You can also directly measure model stability: If you resample your training data, you also see large variations between models trained on slightly different data sets (resampling exchanges some cases against others): the models are unstable. Instead of directly comparing the models, you can also compare predictions of the different "surrogate" models (built on different resampled training sets) for the same test cases.
When you do PCA, you get components that are sorted with decreasing variance. Doing this as a pre-processing step before classification is lead by the assumption that the relevant differences between the classes are the largest variations in the data set, whereas the higher componentents carry only noise. In that scenario, if the right number of PCs is chosen, only noise is cut away while the information (= variance relevant for the classification) is preserved.
So the drop for higher numbers of PCs is not due to lost information but to too high classifier complexity. Your data set probably does not carry enough information to train a stable classifier with the hight complexity corresponding to the high number of PCs.