Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Does it make sense to do PCA before carrying out a Random Forest Classification?

I'm dealing with high dimensional text data, and I want to do feature reduction to help avoid the curse of dimensionality, but don't Random Forests already to some sort of dimension reduction?

share|improve this question
The RF algorithm does not really suffer from high number of predictors since it only take a random subset of them (so called mtry parameter) to build each tree. There is also a recursive feature elimination technique built on top of the RF algorithm (see the varSelRF R package and references therein). It is, however, certainly possible to add an initial data reduction scheme, although it should be part of the cross-validation process. So the question is: do you want to input a linear combination of your features to RF? – chl Jan 10 '13 at 21:03
How many features/dimensions, F? >1K? >10K? Are the features discrete or continuous, e.g. term-frequency, tfidf, similarity metrics, word vectors or what? PCA runtime is quadratic to F. – smci Jun 9 '15 at 22:58
See e.g. Best PCA algorithm for huge number of features? – smci Jun 9 '15 at 23:00
up vote 3 down vote accepted

Leo Brieman wrote that "dimensionality can be a blessing". In general, random forests can run on large data sets without problems. How large is your data? Different fields handle things in different ways depending on subject-matter knowledge. For example, in gene expression studies genes are often discarded based on low variance (no peeking at the outcome) in a process sometimes called non-specific filtering. This can help with the running time on random forests. But it is not required.

Sticking with the gene expression example, sometimes analysts use PCA scores to represent gene expression measurements. The idea is to replace similar profiles with one score that is potentially less messy. Random forests can be run both on the original variables or the PCA scores (a surrogate for the variables). Some have reported better results with this approach, but there are no good comparisons to my knowledge.

In sum, there is no need to do PCA before running RF. But you can. The interpretation could change depending on your goals. If all you want to do is predict, the interpretation may be less important.

share|improve this answer
Thank you for the response. Speed is an issue, more because I have several thousand possible labels in a multi-label problem. The application is classifying a corpus of text data drawn from both twitter and analysts' description of certain events. I'm using tf-idf weighting and the bag of words model. – Maus Jan 10 '13 at 22:04

PCA before random forest can be useful not for dimensionality reduction but to give you data a shape where random forest can perform better.

I am quiet sure that in general if you transform your data with PCA keeping the same dimensionality of the original data you will have a better classification with random forest

share|improve this answer
PCA runtime is quadratic to number of features F, so it's not always cheap. – smci Jun 9 '15 at 22:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.