4
$\begingroup$

I'm trying to get familiar with PCA in relation to other classification methods. I know that if using PCA for preprocessing, then the input data to the machine learning algorithm will rotate. Does this rotation of input affect the classification methods such as KNN, SVM and Random Forest?

Or more precisely are KNN, SVM or Random Forest affected by the transformation in the sense that their classi cation performance may change if trained and tested on the transformed data compared to the original data before using PCA?

Examples of so is more than welcome, so I hopefully can understand it better.

$\endgroup$
2
$\begingroup$

When people talk PCA, they usually means different things: most people doing it for dimension reduction, which means we will lose some information when mapping to lower dimensional space. Let's assume you are not doing dimension reduction and only do rotation on data.

If that is the case, doing PCA is similar to do scaling on the data. It will make certain algorithm works better, at the same time, we lose the original meanings for each feature.

How PCA impact each algorithm is really a big question, that depends on a lot on the algorithm and data. Let's start with decision trees, which is the building block for random forest.

If your data looks like left figure, doing PCA will make things worse: in the right figure, the boundary become oblique, using horizontal and vertical split will be harder to approximate (although we can use oblique tree).

But at the same time, if you are using logistic regression, such operation has no impact.

enter image description here

$\endgroup$
  • $\begingroup$ Shouldn't PCA change the decision tree by changing the terms in the expressions that are used in the decisions (using the principle components instead of the variables)? And as a consequence the boundary will be different in the right images (instead of rotated). $\endgroup$ – Martijn Weterings Jan 19 '18 at 16:32
  • $\begingroup$ @MartijnWeterings I think the boundary in the images is not a decision boundary given by decision tree predictions, the red and black are the actual class labels. Given that, it's a nice example. $\endgroup$ – Matthew Drury Jan 20 '18 at 18:10
  • $\begingroup$ @MatthewDrury thanks for help to clarify I will revise when I have time $\endgroup$ – Haitao Du Jan 20 '18 at 18:25
2
$\begingroup$

I know that if using PCA for preprocessing, then the input data to the machine learning algorithm will rotate.

Not exactly. PCA rotates, and then projects on the features with highest variance. If you only did rotation, how could you call this dimensionality reduction?

Does this rotation of input affect the classification methods such as KNN, SVM and Random Forest?

hxd1011 already tackled trees.

SVM and linear methods - the structure can change after the projection, for example check out this: example of linearly separable data that isn't linearly separable after PCA.

Also I think it this needs to be added to hxd1011's part on logistic regression: it depends whether you use regularization. For example, if you use Lasso or elasticnet, then you'd get different answer: $l_1$ norm is not invariant to rotations. This might be useful to know, since sparsity-inducing norms for linear problems are useful for very high-dimensional problems, and they sort of do feature selection for themselves, so they might be useful in context where you'd need dimensionality reduction before running another model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.