2
$\begingroup$

I write in hopes of understanding an odd behavior of the randomForest package. I am trying to predict a factor y with 9 levels using 8 binary factors X1-X8. I get good accuracy (0.8959), and the following confusion matrix:

                      y
    A   B   C   D   E   F   G   H   I
A  75   0   0   0   0   0   0   0   0
B   0 121   0   5   0   0   0   0   0
C   0   0 156   1   0   0   0   1   0
D   0   0   0   0   0   0   0   0   0
E   1   6   0  73 172   3   0   1   1
F   0   0   0   0   0  90   0   0   0
G   0   0   0   1   0   0  31   0   0
H   0   0   0   1   0   0   0  84   0
I   0   0   0   3   0   0   0   0 106

Notice that RF makes no predictions for row D of the confusion matrix. Now I perform the following experiment: I make a copy of the first column of the predictor matrix, call it "junk", and append it to the predictor matrix. Now randomForest gives improved accuracy (0.9657) and the following confusion matrix:

                  y
    A   B   C   D   E   F   G   H   I
A  73   0   0   0   0   0   0   0   0
B   2 119   0   5   0   0   0   0   0
C   0   2 156   1   0   0   0   1   0
D   1   6   0  73   4   3   0   1   1
E   0   0   0   0 168   0   0   0   0
F   0   0   0   0   0  90   0   0   0
G   0   0   0   1   0   0  31   0   0
H   0   0   0   1   0   0   0  84   0
I   0   0   0   3   0   0   0   0 106

Note that randomForest now makes good predictions for row D of the confusion matrix.

In summary, appending a redundant copy of one of the predictor variables to the predictor matrix improves accuracy of randomForest. Further, it doesn't make much difference which predictor you append. They all give roughly the same accuracy and roughly the same confusion matrix.

I append code and data below. Can someone explain what is happening?

Code:

### Save the file and change the location
setwd("C:\\tmp")

rm(list=ls())
library(randomForest)

# input compressed data and restore the 
# number observed for each row
compressed <- read.csv("compressed.csv")
num <- compressed$NUM
newnum <- rep(1:length(num),num)
dat <- compressed[newnum,2:10]

y <- dat$y
x <- dat[,2:9]

# original data produces bad results 
# for row D of confusion matrix
set.seed(323)
badrf=randomForest(y=y,x=x)
badpred=predict(badrf,newdata=x)
badtable <- table(badpred, y)
badtable
badaccuracy=sum(diag(badtable))/sum(badtable)
badaccuracy

# duplicate, say, x-matrix column 1
ndx <- 1
junk <- x[,ndx]
newx <- cbind(x,junk)

# re-analysis with superfluous new variable 
# gives good results
set.seed(323)
goodrf=randomForest(y=y,x=newx)
goodpred=predict(goodrf,newdata=newx)
goodtable <- table(goodpred, y)
goodtable
goodaccuracy=sum(diag(goodtable))/sum(goodtable)
goodaccuracy

Data:

"NUM","y","X1","X2","X3","X4","X5","X6","X7","X8"
1,"A","NO","NO","NO","NO","NO","NO","NO","NO"
69,"A","NO","NO","YES","NO","NO","NO","NO","NO"
2,"A","NO","NO","YES","NO","NO","NO","NO","YES"
4,"A","NO","YES","YES","NO","NO","NO","NO","NO"
6,"B","NO","NO","NO","NO","NO","NO","NO","NO"
119,"B","NO","NO","NO","NO","NO","NO","NO","YES"
2,"B","YES","NO","NO","NO","NO","NO","NO","YES"
155,"C","YES","NO","NO","NO","NO","NO","NO","NO"
1,"C","YES","YES","NO","NO","NO","NO","NO","NO"
73,"D","NO","NO","NO","NO","NO","NO","NO","NO"
5,"D","NO","NO","NO","NO","NO","NO","NO","YES"
1,"D","NO","NO","NO","NO","NO","NO","YES","NO"
1,"D","NO","NO","NO","NO","YES","NO","NO","NO"
3,"D","NO","YES","NO","NO","NO","NO","NO","NO"
1,"D","YES","NO","NO","NO","NO","NO","NO","NO"
4,"E","NO","NO","NO","NO","NO","NO","NO","NO"
158,"E","NO","NO","NO","NO","NO","YES","NO","NO"
10,"E","YES","NO","NO","NO","NO","YES","NO","NO"
3,"F","NO","NO","NO","NO","NO","NO","NO","NO"
90,"F","NO","NO","NO","YES","NO","NO","NO","NO"
31,"G","NO","NO","NO","NO","NO","NO","YES","NO"
1,"H","NO","NO","NO","NO","NO","NO","NO","NO"
83,"H","NO","NO","NO","NO","YES","NO","NO","NO"
1,"H","NO","YES","NO","NO","YES","NO","NO","NO"
1,"H","YES","NO","NO","NO","YES","NO","NO","NO"
1,"I","NO","NO","NO","NO","NO","NO","NO","NO"
102,"I","NO","YES","NO","NO","NO","NO","NO","NO"
3,"I","NO","YES","NO","NO","NO","NO","NO","YES"
1,"I","NO","YES","NO","NO","NO","NO","YES","NO"
$\endgroup$
  • $\begingroup$ Tune the rf model and it will output similar predictions for the two cases. $\endgroup$ – missuse Nov 15 '18 at 13:03
1
$\begingroup$

For each node in a tree, the random forest algorithm does not pick the best dimension to split on but the best dimension among a (small) sample of them in order to add variance in the prediction.

Here, the column you add must be important and by adding it you make it appear more often in the sample of dimensions among which the random forest chooses. You make it more important in your prediction. And by doing so, you do not decrease too much the variability of your trees.

This behavior could explain your findings

PS: Indeed, ideally, it should be moved to stats.stackexchange

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy