6
$\begingroup$

In Feature Engineering and Selection: A Practical Approach for Predictive Models Kuhn and Johnson mention (chapter 7) that random forest can be used to detect interaction between variables, e.g. x1*x2.

My question is. Can random forest be used to detect squared terms x1^2. If so, how would you do that?

$\endgroup$
8
$\begingroup$

Indeed it can. Here are some simulated data with a squared relationship between the predictor and the response, and the fit from a Random Forest:

rf fit

R code:

nn <- 1e4
set.seed(1)
xx <- runif(nn)
yy <- xx^2+rnorm(nn,0,0.1)

plot(xx,yy,pch=19,cex=0.6,col="lightgray")

library(randomForest)
model <- randomForest(yy~xx)

xx_pred <- seq(0,1,by=.01)
lines(xx_pred,predict(model,newdata=data.frame(xx=xx_pred)),col="red",lwd=2)

As to how the RF does this: remember that it is just a collection of classification and regression trees. Each separate tree (based on a bootstrap sample of the data, and a subset of predictors) will use a different cutoff of the predictor and output a different value for the response for low vs. high predictor values. On average, the fitted reponse for high predictor values will deviate more from the average than for low predictor values, and thus model the nonlinear relationship.

There are also RF implementations that fit linear models on the predictor in the leaves. These can of course also model nonlinearities, by fitting different slopes for different values of the predictor.

$\endgroup$
  • $\begingroup$ Perfect, thank you. One related question. Let's say I want to fit linear model but use random forest to detect interactions and/or squared terms. For interaction effects I can look at variable importance plot to find out if x1 and x2 have interactions. Is there any way to detect those squared terms in a similar way so I can add those to the linear model? $\endgroup$ – Viðar Ingason Feb 17 at 13:33
  • 1
    $\begingroup$ It will depend on how complex your hypothesized relationship is. If you just have one predictor, you can plot it against the response as I did here. If you have something more complicated, you can fit a model with a linear relationship, then plot the residuals against the predictor. If there is a squared relationship, then residuals should show a systematic pattern against the predictor. Alternatively, you could compare the two models (with a squared vs. with only a linear predictor) using ANOVA. $\endgroup$ – Stephan Kolassa Feb 17 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.