16
$\begingroup$

I've seen in a kaggle challenge about digit recognition someone who used PCA before decision tree or other techniques.

I thought it was just for compressing data but he aimed to improve his score.

How can PCA improve score in this case ? Is it because there is less overfitting ?

Improve this question
$\endgroup$
3
  • $\begingroup$ I would be tempted to say that decision trees are harder to train. There are a lot of details in the learning algorithms for them. PCA has a well founded framework to tweak rather than get lost in the details. $\endgroup$
    – Vass
    Commented Apr 3, 2013 at 10:15
  • 1
    $\begingroup$ Dimensionality reduction via PCA can definitely serve as regularization in order to prevent overfitting. E.g. in regression it is known as "principal components regression" and is related to ridge regression. For classification, see e.g. here: Does it make sense to combine PCA and LDA? $\endgroup$
    – amoeba
    Commented Jan 27, 2015 at 16:48
  • $\begingroup$ I think the top answer misses the point of this question (see my comment under it). I'd suggest to read this thread stats.stackexchange.com/questions/141864 and follow the links for the comprehensive discussion. $\endgroup$
    – amoeba
    Commented Jan 25, 2017 at 19:28

4 Answers 4

Reset to default
13
$\begingroup$

Dadi Perlmutter once said: "What is the difference between theory and practice? In theory they are the same while in practice they are different". This is one of those cases.

Methods like Neural Networks often use gradient descent derived methods. In theory if you had infinite number of iterations and retries, the algorithm is going to converge to the same result independent of coordinate system. Neural Networks do not like the "curse of dimensionality" and so using PCA to reduce the dimension of the data can improve speed of convergence and quality of results. The transformation of the data, by centering, rotating and scaling informed by PCA can improve the convergence time and the quality of results.

In theory the PCA makes no difference, but in practice it improves rate of training, simplifies the required neural structure to represent the data, and results in systems that better characterize the "intermediate structure" of the data instead of having to account for multiple scales - it is more accurate.

My guess is that there are analogous reasons that apply to random forests of gradient boosted trees or other similar creatures. (Link)

Improve this answer
$\endgroup$
5
  • 4
    $\begingroup$ Aside: I would be interested to see an accurate and reputable sourcing of your introductory quote. It is variously attributed to several people on the internet, most recognizably, Yogi Berra and Albert Einstein. I have personally heard it from an engineer that is a generation older than Perlmutter and this was long enough ago that it makes me highly doubt that Perlmutter could be the original source. $\endgroup$
    – cardinal
    Commented Apr 3, 2013 at 19:13
  • 3
    $\begingroup$ Cardinal - He is not the original source, but he is the source that I heard it from. I saw him presenting something onstage in 2009. The only thing I retain, 4 years later, is this quote. $\endgroup$
    – EngrStudent
    Commented Apr 3, 2013 at 20:56
  • 1
    $\begingroup$ This conversation is so funny, much more than the quote itself. I had once asked my professor if I can cite a cited quote because the original paper was a phantom one! $\endgroup$
    – KarthikS
    Commented Jun 30, 2016 at 23:45
  • 3
    $\begingroup$ "In theory the PCA makes no difference": This is only the case when all PCs are retained. Usually when people talk about doing PCA prior to some other algorithm they mean that they keep only a small subset of PCs (and the OP wrote about "compressing data" too). Your answer does not cover this possibility at all so I feel like it does not really address the question. $\endgroup$
    – amoeba
    Commented Jan 25, 2017 at 9:28
  • $\begingroup$ Whitening or pinkening transforms are both PCA approaches that retain all the components. The difference is trying to transform the data so that the diagonal of the covariance is more uniform, or less uniform. $\endgroup$
    – EngrStudent
    Commented May 11, 2020 at 19:00
5
$\begingroup$

Disclaimer: I'm usually wrong at things.

Decision trees, by virtue of doing recursive splitting of your samples, with splits being based on a single variable, can only generate decision boundaries parallel to the axes of your co-ordinate system. So by rotating the data to directions of maximum variance/diagonalizing your covariance matrix as best you can, it might be easier to put decision boundaries between your class distributions

That being said, I'm not sure why you'd do PCA (without discarding some of your eigenvectors) before using a neural network model or whatever, because the rotation alone makes no difference - the network can approximate any function through the feature space.

Improve this answer
$\endgroup$
1
$\begingroup$

An insight I gained from Jonathon Shlens' "A Tutorial on Principal Component Analysis": Performing PCA is like choosing a camera angle, to gain the best possible view of the variance to be explained.

So I'm joining user1843053. At proper angle, decision boundaries parallel to the axes of new, rotated coordinate system might make more sense than in original feature space, allowing for better performance of e.g. decision trees, even without discarding "non-principal" dimensions.

Improve this answer
$\endgroup$
0
$\begingroup$

The PCA is a change of variables, using the correlations explained by orthogonal directions.

Removing directions with non-representative corresponding correlation is like removing noise. You will only keep significant datas.

By the way, thanks for the site.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.