Why does my model perform best when I first use PCA with n_components = the number of columns?

Question

I'm training on a dataset with lots of multi-collinearity and after one-hot encoding the categorical variables, there are 90 columns not including the target variable. It seems like this would benefit from dimensionality reduction so I've performed a grid search with 3, 5, 7, 9 and 90 principal components. To my surprise, 90 principal components results in the best performance by far. It even performs better than when I don't use PCA at all.

Why is this? Is there actually any dimensionality reduction happening with 90 components? If not, then why does this give better results than when PCA is excluded entirely?

I'm passing the output of PCA to a gradient boosted classifier. By "better performance" Im referring to a combination of accuracy, recall and AUC.

Answer 1

Using all the components amounts to having rotated your data. For a tree-based model, this can have an effect: the tree splits are along axis-aligned planes, and so rotation changes what splits are available. The model can approximate non-axis-aligned splits by making many axis-aligned splits, but making it easier to find useful splits (by rotation or otherwise) can help the greedy tree-building process find useful information. (I wouldn't expect the improvement to be very large, but this can depend on your hyperparameters and the nature of the principal components.)

Answer 2

If you are using all 90 components then you are not really doing any dimension reduction. After all, you started with 90 covariates and you are now using 90 principal components. Now when you say better results, what do you mean exactly by this? By what metric are you measuring this. For example, do you mean that you are explaining all the variance in the data? This Is of course true because you are using all the components.