PCA not helping in very high dimensional regression using KNN I am using sklearn.neighbors.KNeighborsRegressor() to train over a set of very high dimensional input (721 dimensions) and continous output (a regression problem). My sample size is 2500. I heard of the curse of dimensionality so I tried implementing PCA to reduce the amount of features. The image below shows the $R^2$ score vs # neighbors for  different number of features taken. By the way, these $R^2$ scores are the cross-validation scores reported by GridSearchCV() and not an out-of-sample score, but they should be similar I guess. The little triangle marks indicate the optimal scores. 

Now, I would have expected that lowering the amount of features would get me better scores, as it would help with the curse of dimensionality, but this is not the case. In fact, the more components you take, the better $R^2$ score I get. To be completely honest, the GridSearchCV() reported that the best parameters were n_components = 13, n_neighbors = 7, but as you can see in the picture, it might have been just a numerical luck. According to the plot, the performances I get are pretty much the same after 7 features. I would have expected the curves to be higher until some point, and then be lower due to the curse of dimensionality. Why is it not the case?
EDIT: Added a simpler plot of the cross-validation score vs PCA components. As mentioned, after some 10 components, the best scores are pretty much constant.

EDIT2: Added scree plot. 

 A: The primary goal of PCA is to reduce the dimensionality of the data where the effect can be a reduction in computation, reduced sparsity, and in general easier to handle. PCA does not necessarily improve the model as, @usεr11852 pointed out, and since PCA doesn't take into account the modeling objective it could summarize irrelevant information. 
Each principle component added will in general increase the $R^2$ score. As more and more variance will be explained.
The first knee of the scree plot is a rule of thumb for capturing a significant amount of the variance. Eyeballing your scree plot I'd say it is close to 5-6. I see from your first plot that five PCs seem to be the threshold where a lot of variance is lost. Results with seven or more components are about the same.
In summary it is possible that PCA will do no better than the entire set of original features.
A: @MattL's answer is one possibility. Here's another - which in my experience is a rather common mistake:
When cross-validating, does your CV routine build also a new PCA model for each split? If not, you're quite likely to overfit to the one PCA projection you calculated.
