I write in hopes of understanding an odd behavior of the randomForest package. I am trying to predict a factor y with 9 levels using 8 binary factors X1-X8. I get good accuracy (0.8959), and the following confusion matrix:
y
A B C D E F G H I
A 75 0 0 0 0 0 0 0 0
B 0 121 0 5 0 0 0 0 0
C 0 0 156 1 0 0 0 1 0
D 0 0 0 0 0 0 0 0 0
E 1 6 0 73 172 3 0 1 1
F 0 0 0 0 0 90 0 0 0
G 0 0 0 1 0 0 31 0 0
H 0 0 0 1 0 0 0 84 0
I 0 0 0 3 0 0 0 0 106
Notice that RF makes no predictions for row D of the confusion matrix. Now I perform the following experiment: I make a copy of the first column of the predictor matrix, call it "junk", and append it to the predictor matrix. Now randomForest gives improved accuracy (0.9657) and the following confusion matrix:
y
A B C D E F G H I
A 73 0 0 0 0 0 0 0 0
B 2 119 0 5 0 0 0 0 0
C 0 2 156 1 0 0 0 1 0
D 1 6 0 73 4 3 0 1 1
E 0 0 0 0 168 0 0 0 0
F 0 0 0 0 0 90 0 0 0
G 0 0 0 1 0 0 31 0 0
H 0 0 0 1 0 0 0 84 0
I 0 0 0 3 0 0 0 0 106
Note that randomForest now makes good predictions for row D of the confusion matrix.
In summary, appending a redundant copy of one of the predictor variables to the predictor matrix improves accuracy of randomForest. Further, it doesn't make much difference which predictor you append. They all give roughly the same accuracy and roughly the same confusion matrix.
I append code and data below. Can someone explain what is happening?
Code:
### Save the file and change the location
setwd("C:\\tmp")
rm(list=ls())
library(randomForest)
# input compressed data and restore the
# number observed for each row
compressed <- read.csv("compressed.csv")
num <- compressed$NUM
newnum <- rep(1:length(num),num)
dat <- compressed[newnum,2:10]
y <- dat$y
x <- dat[,2:9]
# original data produces bad results
# for row D of confusion matrix
set.seed(323)
badrf=randomForest(y=y,x=x)
badpred=predict(badrf,newdata=x)
badtable <- table(badpred, y)
badtable
badaccuracy=sum(diag(badtable))/sum(badtable)
badaccuracy
# duplicate, say, x-matrix column 1
ndx <- 1
junk <- x[,ndx]
newx <- cbind(x,junk)
# re-analysis with superfluous new variable
# gives good results
set.seed(323)
goodrf=randomForest(y=y,x=newx)
goodpred=predict(goodrf,newdata=newx)
goodtable <- table(goodpred, y)
goodtable
goodaccuracy=sum(diag(goodtable))/sum(goodtable)
goodaccuracy
Data:
"NUM","y","X1","X2","X3","X4","X5","X6","X7","X8"
1,"A","NO","NO","NO","NO","NO","NO","NO","NO"
69,"A","NO","NO","YES","NO","NO","NO","NO","NO"
2,"A","NO","NO","YES","NO","NO","NO","NO","YES"
4,"A","NO","YES","YES","NO","NO","NO","NO","NO"
6,"B","NO","NO","NO","NO","NO","NO","NO","NO"
119,"B","NO","NO","NO","NO","NO","NO","NO","YES"
2,"B","YES","NO","NO","NO","NO","NO","NO","YES"
155,"C","YES","NO","NO","NO","NO","NO","NO","NO"
1,"C","YES","YES","NO","NO","NO","NO","NO","NO"
73,"D","NO","NO","NO","NO","NO","NO","NO","NO"
5,"D","NO","NO","NO","NO","NO","NO","NO","YES"
1,"D","NO","NO","NO","NO","NO","NO","YES","NO"
1,"D","NO","NO","NO","NO","YES","NO","NO","NO"
3,"D","NO","YES","NO","NO","NO","NO","NO","NO"
1,"D","YES","NO","NO","NO","NO","NO","NO","NO"
4,"E","NO","NO","NO","NO","NO","NO","NO","NO"
158,"E","NO","NO","NO","NO","NO","YES","NO","NO"
10,"E","YES","NO","NO","NO","NO","YES","NO","NO"
3,"F","NO","NO","NO","NO","NO","NO","NO","NO"
90,"F","NO","NO","NO","YES","NO","NO","NO","NO"
31,"G","NO","NO","NO","NO","NO","NO","YES","NO"
1,"H","NO","NO","NO","NO","NO","NO","NO","NO"
83,"H","NO","NO","NO","NO","YES","NO","NO","NO"
1,"H","NO","YES","NO","NO","YES","NO","NO","NO"
1,"H","YES","NO","NO","NO","YES","NO","NO","NO"
1,"I","NO","NO","NO","NO","NO","NO","NO","NO"
102,"I","NO","YES","NO","NO","NO","NO","NO","NO"
3,"I","NO","YES","NO","NO","NO","NO","NO","YES"
1,"I","NO","YES","NO","NO","NO","NO","YES","NO"
