# PCA not helping in very high dimensional regression using KNN

I am using sklearn.neighbors.KNeighborsRegressor() to train over a set of very high dimensional input (721 dimensions) and continous output (a regression problem). My sample size is 2500. I heard of the curse of dimensionality so I tried implementing PCA to reduce the amount of features. The image below shows the $R^2$ score vs # neighbors for different number of features taken. By the way, these $R^2$ scores are the cross-validation scores reported by GridSearchCV() and not an out-of-sample score, but they should be similar I guess. The little triangle marks indicate the optimal scores.

Now, I would have expected that lowering the amount of features would get me better scores, as it would help with the curse of dimensionality, but this is not the case. In fact, the more components you take, the better $R^2$ score I get. To be completely honest, the GridSearchCV() reported that the best parameters were n_components = 13, n_neighbors = 7, but as you can see in the picture, it might have been just a numerical luck. According to the plot, the performances I get are pretty much the same after 7 features. I would have expected the curves to be higher until some point, and then be lower due to the curse of dimensionality. Why is it not the case?

EDIT: Added a simpler plot of the cross-validation score vs PCA components. As mentioned, after some 10 components, the best scores are pretty much constant.

• Can you share a scree-plot from your PCA? BTW, $R^2$ values $\sim 0.70$ are generally very good. Lowering the amount for features used will get a you a better scores if the features excluded are not useful. It is not guaranteed that you will have improved performance. Finally you are looking at $R^2$; this metric does not penalize the number of features used so the fact that it does not get lower is not surprising at all (if you used adjusted $R^2$ that would be surprising...) – usεr11852 Feb 20 '16 at 4:59
• Sorry for late reply. I added it now. – Michael Feb 22 '16 at 13:05
• Show the cumulative explained variance that corresponds to your scree plot. For instance, how many PCs you need to explain 90% of the total variance? – Aksakal Feb 22 '16 at 14:09
• @Aksakal: I guess ~8 based on that screeplot. – usεr11852 Feb 22 '16 at 17:58

The primary goal of PCA is to reduce the dimensionality of the data where the effect can be a reduction in computation, reduced sparsity, and in general easier to handle. PCA does not necessarily improve the model as, @usεr11852 pointed out, and since PCA doesn't take into account the modeling objective it could summarize irrelevant information.

Each principle component added will in general increase the $R^2$ score. As more and more variance will be explained.

The first knee of the scree plot is a rule of thumb for capturing a significant amount of the variance. Eyeballing your scree plot I'd say it is close to 5-6. I see from your first plot that five PCs seem to be the threshold where a lot of variance is lost. Results with seven or more components are about the same.

In summary it is possible that PCA will do no better than the entire set of original features.

@MattL's answer is one possibility. Here's another - which in my experience is a rather common mistake:

When cross-validating, does your CV routine build also a new PCA model for each split? If not, you're quite likely to overfit to the one PCA projection you calculated.

• +1 good point that is often missed. OP has 2500 samples so $k=10$ would still mean that the training set gets ~2250 samples but overfitting is a possibility. As I said thought, he uses $R^2$ and that does not penalize for more variable so I do not see the reason why a significant drop off should be there... – usεr11852 Feb 22 '16 at 18:03
• Yes, PCA is inside the CV pipeline. If 3-fold CV, it will take 1 fold out, perform scaling+pca+fitting in the remaining 2 folds, and so on. – Michael Feb 23 '16 at 11:17