Do PCA affect different classification methods?

Question

I'm trying to get familiar with PCA in relation to other classification methods. I know that if using PCA for preprocessing, then the input data to the machine learning algorithm will rotate. Does this rotation of input affect the classification methods such as KNN, SVM and Random Forest?

Or more precisely are KNN, SVM or Random Forest affected by the transformation in the sense that their classication performance may change if trained and tested on the transformed data compared to the original data before using PCA?

Examples of so is more than welcome, so I hopefully can understand it better.

Answer 1

When people talk PCA, they usually means different things: most people doing it for dimension reduction, which means we will lose some information when mapping to lower dimensional space. Let's assume you are not doing dimension reduction and only do rotation on data.

If that is the case, doing PCA is similar to do scaling on the data. It will make certain algorithm works better, at the same time, we lose the original meanings for each feature.

How PCA impact each algorithm is really a big question, that depends on a lot on the algorithm and data. Let's start with decision trees, which is the building block for random forest.

If your data looks like left figure, doing PCA will make things worse: in the right figure, the boundary become oblique, using horizontal and vertical split will be harder to approximate (although we can use oblique tree).

But at the same time, if you are using logistic regression, such operation has no impact.

Answer 2

I know that if using PCA for preprocessing, then the input data to the machine learning algorithm will rotate.

Not exactly. PCA rotates, and then projects on the features with highest variance. If you only did rotation, how could you call this dimensionality reduction?

Does this rotation of input affect the classification methods such as KNN, SVM and Random Forest?

hxd1011 already tackled trees.

SVM and linear methods - the structure can change after the projection, for example check out this: example of linearly separable data that isn't linearly separable after PCA.

Also I think it this needs to be added to hxd1011's part on logistic regression: it depends whether you use regularization. For example, if you use Lasso or elasticnet, then you'd get different answer: $l_1$ norm is not invariant to rotations. This might be useful to know, since sparsity-inducing norms for linear problems are useful for very high-dimensional problems, and they sort of do feature selection for themselves, so they might be useful in context where you'd need dimensionality reduction before running another model.