15
$\begingroup$

I have a regression dataset where the features are on the order of ~ 400 variables and the dataset itself is around 300 samples. I tried to use Random Forest Regression (RFR) on the dataset and used either out-of-bag (oob) score or k-fold cv score to judge its performance. The kind of behavior I see right now that I'm trying to make sense of is that if I directly use RFR, no matter how many trees I use or what kind of parameter tuning I incorporate, I won't get a good performance, whereas if I incorporate a PCA before RFR, I can run a grid search for the number of PCs before RFR and at around 8 or 9 PCs, the processing flow could provide a somewhat descent score. The score would rise and fall around this "optimal PC number" when I sweep the number of PCs.

I'm trying to make sense of this behavior as I tried to use the same processing flow on a couple of toy datasets I found and usually with or without PCA won't change much for RFR performance. One of the concerns I had is that my dataset is a very noisy dataset, and most of the regression methods I tried so far won't provide much good performance except for this PCA-RFR flow. So I'm not sure if this is a garbage-in-garbage-out situation where this PCA-RFR thing just somehow overfit my dataset. On the other hand, my features are quite collinear to each other and I don't have that many data to train my model, so it kind of make sense that a PCA-preprocessing can help de-noise the dataset a bit and may also help to reduce the overfit of my training set with a smaller set of "reduced features", but RFR is kind of new to me so I'm not aware if there is any theory behind all these.

If anyone has seen this before and have a good explanation or have any reference paper on PCA-RFR behavior, please let me know and I'd be very much grateful.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ PCA + some kind of regression model (typically logistic) is a widely used pipeline. Basically, PCA allows you to reduce the number of you predictors, and to decorrelate them. This is why it improves the performance of the model downstream. Although in the case of RF (compared, say, to linear or logistic regression), the gains should be minimal because off-the-shelf, the model deals already well with correlated predictors and high dimensional data sets. $\endgroup$
    – Antoine
    Commented Jan 30, 2017 at 13:43
  • $\begingroup$ (+1) We have a very similar question about PCA+RF stats.stackexchange.com/questions/47457 but I don't see satisfactory answers there. I voted to close this question as a duplicate of a general thread on PCA + predictive model. Not sure if there is anything specific about RF here. $\endgroup$
    – amoeba
    Commented Jan 30, 2017 at 13:50
  • $\begingroup$ Optimizing an equation with 1 variable almost always gives better results than optimizing an equation with 10 variables, that is the well know dimensionality curse. When you do PCA you are essentially reducing the amount of variables but still describing the same data. $\endgroup$
    – Ander Biguri
    Commented Jan 30, 2017 at 16:18

2 Answers 2

Reset to default
16
$\begingroup$

Using Random Forest in a dataset as the one you described has two major problems:

  1. Random Forest does not perform well when features are monotonic transformations of other features (this makes the trees of the forest less independent from each other).

  2. The same happens when you have more features than samples: random forest will probably overfit the dataset, and you will have a poor out of bag performance.

When using PCA you get rid of these two problems that are lowering the performance of Random Forest:

  1. you reduce the number of features.
  2. you get rid of collinear features. (all collinear features will end up in a single PCA component).
Improve this answer
$\endgroup$
8
$\begingroup$

I think you just answered yourself. In general RF are not good in high dimensional settings or when you have more features than samples, therefore reducing your features from 400 to 8 will help, especially if you have lot's of noisy collinear features. You have also less chance to overfit in this case, but beware of double-dipping and model selection bias. So that you run lot's of models and choose the best one, which might be best just by chance and wouldn't generalize on unseen data.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.