17
$\begingroup$

I'm doing regression using Random Forests for predicting prices based on several attributes. Code is written in Python using Scikit-learn.

How do you decide whether you should transform your variables using exp/log before using it to fit the regression model? Is it necessary when using an Ensemble approach such as Random Forest?

Improve this question
$\endgroup$
4
  • $\begingroup$ possible duplicate of In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values? $\endgroup$
    – user603
    May 11, 2013 at 9:13
  • 5
    $\begingroup$ I think this is a bit different from that question, due to the random forest/ensemble aspect, but the question could perhaps be reworded after looking at the other post. $\endgroup$
    – Peter Flom
    May 11, 2013 at 12:12
  • $\begingroup$ @PeterFlom could you help me reword the question? I am not proficient enough in this field :) $\endgroup$
    – Nyxynyx
    May 11, 2013 at 17:56
  • $\begingroup$ The way most people use the term "ensemble" RF is just 1 potential input to an ensemble. $\endgroup$
    – Hack-R
    Aug 5, 2018 at 19:53

2 Answers 2

Reset to default
22
$\begingroup$

The way Random Forests are built is invariant to monotonic transformations of the independent variables. Splits will be completely analogous. If you are just aiming for accuracy you will not see any improvement in it. In fact, since Random Forests are able to find complex non-linear (Why are you calling this linear regression?) relations and variable interactions on the fly, if you transform your independent variables you may smooth out the information that allows this algorithm to do this properly.

Sometimes Random Forests are not treated as a black box and used for inference. For example, you can interpret the variable importance measures that it provides, or calculate some sort of marginal effects of your independent variable on your dependent variable. This is usually visualized as partial dependence plots. I'm pretty sure this last thing is highly influenced by the scale of the variables, which is a problem when trying to obtain information of a more descriptive nature from Random Forests. In this case it might help you to transform your variables (standardize), which could make partial dependence plots comparable. Not completely sure on this, will have to think on it.

Not long ago I tried to predict count data using a Random Forest, regressing on the square root and the natural log of the dependant variable helped a bit, not much, and not enough to let me keep the model.

Some packages with which you may use random forests for inference:

https://uc-r.github.io/lime

https://cran.r-project.org/web/packages/randomForestExplainer/index.html

https://pbiecek.github.io/DALEX_docs/2-2-useCaseApartmetns.html

Improve this answer
$\endgroup$
7
$\begingroup$

Echoing @JEquihua, Random Forest prediction accuracy won't improve.

Also note, if you keep both the original predictor and the transformed predictor (as is often done in linear regression), you may cause problems. That's because RF randomly chooses a subset of the variables to grow each tree, and you've essentially put the transformed variable in twice. If it's a strong predictor, it will get used, and your random forests won't be as uncorrelated as they might have been, leading to higher variance.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Putting in both a transformed and untransformed predictor into any model is a terrible idea. They are perfectly collinear and will certainly mess with your model, whether it be random forest or linear regression. $\endgroup$
    – mkt
    Sep 5, 2019 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.