Random forest validation worse than one of its variables by itself

Question

I am running a random forest in SAS using 6 variables, one of them being a score that works very well on its own. When I train the forest and validate on a test set, I'm seeing the following in terms of rank ordering the dependent var:

RANDOM FOREST VALIDATION

scorerank   Bads    Total   BadRate
0           288    3878     0.0742650851
1           407    3879     0.1049239495
2           520    3878     0.134089737
3           602    3879     0.1551946378
4           729    3878     0.1879834966

So there is a clear separation in the 5 groups of rank ordered probabilities. However, when I use just the score on the same validation set I see this.

SCORE VALIDATION

scorerank   Bads    Total   BadRate
 0          789     3891    0.2027756361
 1          616     3806    0.161849711
 2          488     3766    0.1295804567
 3          397     4213    0.0942321386
 4          256     3716    0.0688912809

The direction of the ranking is reversed but not my concern as that is expected since a lower score is worse than a higher score. What is concerning is that the gap between the best and worst group is higher, indicating better separation.

So conceptually how can a single variable in a random forest outperform the forest? Is this to be expected sometimes? Is there something I can tune in the model?

I am using proc hpforest in SAS.

Answer 1

You can see how this sort of thing can happen by looking at an extreme case. Sorry I can't show you SAS code, but here is a dummy example in R:

library(randomForest)
library(MASS)
N <- 1000 #1000 observations
p <- 10 #10 variables
X <- mvrnorm(N, rep(0, p), diag(rep(1, p))) #The variables are all normal and independent
y <- X[,1] + rnorm(N, sd = .1) #the outcome is a function of the first of the 10 variables plus random noise

using all the variables, looking at 1/3rd of them by default at each split:

rf1 <- randomForest(y = y, x = X)

using only the good variable:

rf2 <- randomForest(y = y, x = X[,1, drop = FALSE])

using all the variables, but tying all the variables at each split

rf3 <- randomForest(y = y, x = X, mtry = 10)

1> rf1

Call:
 randomForest(x = X, y = y) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 3

          Mean of squared residuals: 0.04463831
                    % Var explained: 95.32
1> rf2

Call:
 randomForest(x = X[, 1, drop = FALSE], y = y) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 1

          Mean of squared residuals: 0.01377959
                    % Var explained: 98.56
1> rf3

Call:
 randomForest(x = X, y = y, mtry = 10) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 10

          Mean of squared residuals: 0.01192359
                    % Var explained: 98.75

So you see that random forest is not completely invulnerable to noise variables. Why does this happen? Because in each split of each tree, the method of random subspaces will select from among the available variables and the useful ones may not be among that split!

When we set the algo to look at each variable at each split, we get the same performance as when we only look at the good one.