# Why does adding a redundant predictor to randomForest improve prediction?

I write in hopes of understanding an odd behavior of the randomForest package. I am trying to predict a factor y with 9 levels using 8 binary factors X1-X8. I get good accuracy (0.8959), and the following confusion matrix:

                      y
A   B   C   D   E   F   G   H   I
A  75   0   0   0   0   0   0   0   0
B   0 121   0   5   0   0   0   0   0
C   0   0 156   1   0   0   0   1   0
D   0   0   0   0   0   0   0   0   0
E   1   6   0  73 172   3   0   1   1
F   0   0   0   0   0  90   0   0   0
G   0   0   0   1   0   0  31   0   0
H   0   0   0   1   0   0   0  84   0
I   0   0   0   3   0   0   0   0 106


Notice that RF makes no predictions for row D of the confusion matrix. Now I perform the following experiment: I make a copy of the first column of the predictor matrix, call it "junk", and append it to the predictor matrix. Now randomForest gives improved accuracy (0.9657) and the following confusion matrix:

                  y
A   B   C   D   E   F   G   H   I
A  73   0   0   0   0   0   0   0   0
B   2 119   0   5   0   0   0   0   0
C   0   2 156   1   0   0   0   1   0
D   1   6   0  73   4   3   0   1   1
E   0   0   0   0 168   0   0   0   0
F   0   0   0   0   0  90   0   0   0
G   0   0   0   1   0   0  31   0   0
H   0   0   0   1   0   0   0  84   0
I   0   0   0   3   0   0   0   0 106


Note that randomForest now makes good predictions for row D of the confusion matrix.

In summary, appending a redundant copy of one of the predictor variables to the predictor matrix improves accuracy of randomForest. Further, it doesn't make much difference which predictor you append. They all give roughly the same accuracy and roughly the same confusion matrix.

I append code and data below. Can someone explain what is happening?

Code:

### Save the file and change the location
setwd("C:\\tmp")

rm(list=ls())
library(randomForest)

# input compressed data and restore the
# number observed for each row
num <- compressed$NUM newnum <- rep(1:length(num),num) dat <- compressed[newnum,2:10] y <- dat$y
x <- dat[,2:9]

# original data produces bad results
# for row D of confusion matrix
set.seed(323)

# duplicate, say, x-matrix column 1
ndx <- 1
junk <- x[,ndx]
newx <- cbind(x,junk)

# re-analysis with superfluous new variable
# gives good results
set.seed(323)
goodrf=randomForest(y=y,x=newx)
goodpred=predict(goodrf,newdata=newx)
goodtable <- table(goodpred, y)
goodtable
goodaccuracy=sum(diag(goodtable))/sum(goodtable)
goodaccuracy


Data:

"NUM","y","X1","X2","X3","X4","X5","X6","X7","X8"
1,"A","NO","NO","NO","NO","NO","NO","NO","NO"
69,"A","NO","NO","YES","NO","NO","NO","NO","NO"
2,"A","NO","NO","YES","NO","NO","NO","NO","YES"
4,"A","NO","YES","YES","NO","NO","NO","NO","NO"
6,"B","NO","NO","NO","NO","NO","NO","NO","NO"
119,"B","NO","NO","NO","NO","NO","NO","NO","YES"
2,"B","YES","NO","NO","NO","NO","NO","NO","YES"
155,"C","YES","NO","NO","NO","NO","NO","NO","NO"
1,"C","YES","YES","NO","NO","NO","NO","NO","NO"
73,"D","NO","NO","NO","NO","NO","NO","NO","NO"
5,"D","NO","NO","NO","NO","NO","NO","NO","YES"
1,"D","NO","NO","NO","NO","NO","NO","YES","NO"
1,"D","NO","NO","NO","NO","YES","NO","NO","NO"
3,"D","NO","YES","NO","NO","NO","NO","NO","NO"
1,"D","YES","NO","NO","NO","NO","NO","NO","NO"
4,"E","NO","NO","NO","NO","NO","NO","NO","NO"
158,"E","NO","NO","NO","NO","NO","YES","NO","NO"
10,"E","YES","NO","NO","NO","NO","YES","NO","NO"
3,"F","NO","NO","NO","NO","NO","NO","NO","NO"
90,"F","NO","NO","NO","YES","NO","NO","NO","NO"
31,"G","NO","NO","NO","NO","NO","NO","YES","NO"
1,"H","NO","NO","NO","NO","NO","NO","NO","NO"
83,"H","NO","NO","NO","NO","YES","NO","NO","NO"
1,"H","NO","YES","NO","NO","YES","NO","NO","NO"
1,"H","YES","NO","NO","NO","YES","NO","NO","NO"
1,"I","NO","NO","NO","NO","NO","NO","NO","NO"
102,"I","NO","YES","NO","NO","NO","NO","NO","NO"
3,"I","NO","YES","NO","NO","NO","NO","NO","YES"
1,"I","NO","YES","NO","NO","NO","NO","YES","NO"

• Tune the rf model and it will output similar predictions for the two cases. – missuse Nov 15 '18 at 13:03