15
$\begingroup$

I'm doing regression using Random Forests for predicting prices based on several attributes. Code is written in Python using Scikit-learn.

How do you decide whether you should transform your variables using exp/log before using it to fit the regression model? Is it necessary when using an Ensemble approach such as Random Forest?

$\endgroup$
19
$\begingroup$

The way Random Forests are built is invariant to monotonic transformations of the independent variables. Splits will be completely analogous. If you are just aiming for accuracy you will not see any improvement in it. In fact, since Random Forests are able to find complex non-linear (Why are you calling this linear regression?) relations and variable interactions on the fly, if you transform your independent variables you may smooth out the information that allows this algorithm to do this properly.

Sometimes Random Forests are not treated as a black box and used for inference. For example, you can interpret the variable importance measures that it provides, or calculate some sort of marginal effects of your independent variable on your dependent variable. This is usually visualized as partial dependence plots. I'm pretty sure this last thing is highly influenced by the scale of the variables, which is a problem when trying to obtain information of a more descriptive nature from Random Forests. In this case it might help you to transform your variables (standardize), which could make partial dependence plots comparable. Not completely sure on this, will have to think on it.

Not long ago I tried to predict count data using a Random Forest, regressing on the square root and the natural log of the dependant variable helped a bit, not much, and not enough to let me keep the model.

Some packages with which you may use random forests for inference:

https://uc-r.github.io/lime

https://cran.r-project.org/web/packages/randomForestExplainer/index.html

https://pbiecek.github.io/DALEX_docs/2-2-useCaseApartmetns.html

| cite | improve this answer | |
$\endgroup$
7
$\begingroup$

Echoing @JEquihua, Random Forest prediction accuracy won't improve.

Also note, if you keep both the original predictor and the transformed predictor (as is often done in linear regression), you may cause problems. That's because RF randomly chooses a subset of the variables to grow each tree, and you've essentially put the transformed variable in twice. If it's a strong predictor, it will get used, and your random forests won't be as uncorrelated as they might have been, leading to higher variance.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Putting in both a transformed and untransformed predictor into any model is a terrible idea. They are perfectly collinear and will certainly mess with your model, whether it be random forest or linear regression. $\endgroup$ – mkt - Reinstate Monica Sep 5 '19 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.