I'm aware that linear transformations of individual features do not affect Random Forest.

But what if features were linearly recombined to construct a new feature?

I do the following: I take each sample, i.e. each training example, and linearly combine the features for that training example. The result of this combination is a single value, and this single value becomes another feature for this training example. For all of the samples, I do the same: Linearly combining all of the values for the features, creating this new feature. The resulting data matrix of course has an extra column for this extra feature.

  • Does this impact the results of a Random Forest?

  • It is not a linear/monotonic transformation of any one of the features, but rather it is a linear combination of all of the features for a single training example.


Yes multivariate linear re-combinations of variables do affect the possible RF model structure. One could e.g. include the first principal component(PCA) as new feature before training, though certainly not guaranteed to improve model performance. But linear combinations can in some occasions increase the cross validated prediction performance. RF models do fit interactions, but not very efficiently. A tree model interpolate in feature space parallel(not diagonal) to any feature axis.

Conventional tree models only split by one feature, thus a split in feature space is parallel to this feature and orthogonal to every other feature. With a linear recombination you allow to make splits in the old feature space in a new direction, if this new direction happens to align with a steep gradient of a strong interaction, your RF model can reduce its bias.

More general rotation forest rotate partly randomly partly with PCA the entire coordinate system of the feature space before growing each tree. Rotation forest can fit $y=x1*x2*x3*x4$ which is nearly impossible for regular RF, but so can various boosting algorithms. I wrote a RF-variant, which would besides regular features, also try random rotations of features in each node. Successful random rotations was inherited by daughter nodes. The variant was better than RF for complex low noise structures, but not better then gradient boosting machine etc. PPforest use linear discriminant analysis to rotate inside each node. If the data structure consist of nice additive main effects, it is a bad idea to rotate the feature space. Then, main effects will degrade to interaction effects.

enter image description here

To illustrate I made an simulation. 2000 points drawn from $y=x_1*x_2$. RF can fit this structure, but not without some bias. Including an extra variable $x_3=x_1+x_2$ allow what seems as diagonal splits in the original feature space. Notice diagonol splits in right decision tree model(rf123), whereas no diagonal split in middle plot(rf12)


#make data
X  = data.frame(replicate(2,10*(rnorm(2000)-.7)))
y = X[,1]*X[,2]
X[,3] = X[,1] + X[,2] # X3 is a derived variable

#plot the data structure

#train model with/without derived X3, only one tree
rf12  = randomForest(X[,1:2],y,ntree=1)
rf123 = randomForest(X[,1:3],y,ntree=1)

#make plotter function
plotSurf = function(model,X,i.var=1:2,gridLines=150,...) {
  ranges =lapply(X[,i.var],range)
  coordinates = lapply(ranges,function(x) seq(x[1],x[2],le=gridLines)) 
  Xgrid = data.frame(expand.grid(coordinates))
  Xgrid$X3 = apply(Xgrid,1,sum) # reconstruct helper variable
  names(Xgrid) = names(X)
  predGrid =predict(model,Xgrid)
  persp3d(x = coordinates$X1,
          y = coordinates$X2,
          z = predGrid,
          col = fcol(as.matrix(predGrid)),

plotSurf(rf12 ,X,main="rf12",alpha=.6)
  • 1
    $\begingroup$ Wow, great explanation and example. The reason that I do not use the Rotational Forest and the PPForest are two reasons: 1) I worry about interpretability. 2) I worry about overfitting, especially not enough examples are present. Any comments on reducing these fears? Thanks. $\endgroup$ – makansij Dec 21 '15 at 0:08
  • $\begingroup$ Well RF and rotation forest will for a large part be equally difficult to interpret. The effective model structure, though high-dimensional and non-linear, are typically fairly simple (only low order interactions) and there are tools to comprehend a given model fit. By my opinion the individual trees of the forest should mainly be visualized to discuss the algorithm not the effective learned model structure nor the data. Humans cannot comprehend the average model of 500 trees. $\endgroup$ – Soren Havelund Welling Dec 21 '15 at 12:25
  • $\begingroup$ Rotation does as RF not overfit badly. Expected prediction performance by any model and possible overfitting is empirically estimated by cross validation. $\endgroup$ – Soren Havelund Welling Dec 21 '15 at 12:28
  • $\begingroup$ I preach this solution to interpret RF: stats.stackexchange.com/questions/172761/… $\endgroup$ – Soren Havelund Welling Dec 21 '15 at 12:29

Example in 2 dimension: Consider if your true region of consideration is x+y>0 Now if your random forest just gets to see 2 variables, it shall have a deciasion tree as : if (x<1): if (y>1): if (x<-1): ... ... Basically if you see in a plot, it comes across a step function. However, had there been created a feature x+y, the Classifier would have been simple. (One step decision tree)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.