How to decide on optimum number of components for KNN classification

Question

I am doing KNN classification using PCA method. For this, I first did PCA on train data and then predicted components for test set using train PCA. So my train PCA plot looks like this:

I decided to go for first 15 components for test set prediction on KNN classifier. Based on this, I got the following error estimates (plot below). The accuracy estimate (~30%) is acceptable in this case because the dataset has multidimensional responses (i.e response A,B and C grouped together). The error estimate from KNN looks like this:

So the questions I have are:

1. I have 99 percent variability explained by 15 components and I decided to select 15 components for KNN training. Is there a minimum/maximum variability I should consider for this?
2. What's the disadvantage of selecting too many or too few components? Can you not select all components?

Answer 1

In general there is no consensus about minimum / maximum variability. It is also discussed in this question. It depends on what you are trying to predict or regression against, and where the important predictor lies.

As commented in Principal Component Analysis by Joliffe, section 8.2:

Deletion based solely on variance is an attractive and simple strategy, and Property A7 of Section 3.1 gives it, at first sight, an added respectability. However, low variance for a component does not necessarily imply that the corresponding component is unimportant in the regression model. For example, Kung and Sharif (1980) give an example from meteorology where, in a regression of monsoon onset dates on all of the (ten) PCs, the most important PCs for prediction are, in decreasing order of importance, the eighth, second and tenth

If you take too many PCs, you might end up including what are basically noise variables and your model might not be so good.

One thing you can do (also suggested for regression in Joliffe) is to use cross-validation to see how increasing or decreasing the PCs affect the model. You can work with some k value for kNN and see whether it does get less accurate with more PCs.

There's also a recent review on applications of PCA and might be useful for you