2
$\begingroup$

I am running a random forest in SAS using 6 variables, one of them being a score that works very well on its own. When I train the forest and validate on a test set, I'm seeing the following in terms of rank ordering the dependent var:

RANDOM FOREST VALIDATION

scorerank   Bads    Total   BadRate
0           288    3878     0.0742650851
1           407    3879     0.1049239495
2           520    3878     0.134089737
3           602    3879     0.1551946378
4           729    3878     0.1879834966

So there is a clear separation in the 5 groups of rank ordered probabilities. However, when I use just the score on the same validation set I see this.

SCORE VALIDATION

scorerank   Bads    Total   BadRate
 0          789     3891    0.2027756361
 1          616     3806    0.161849711
 2          488     3766    0.1295804567
 3          397     4213    0.0942321386
 4          256     3716    0.0688912809

The direction of the ranking is reversed but not my concern as that is expected since a lower score is worse than a higher score. What is concerning is that the gap between the best and worst group is higher, indicating better separation.

So conceptually how can a single variable in a random forest outperform the forest? Is this to be expected sometimes? Is there something I can tune in the model?

I am using proc hpforest in SAS.

$\endgroup$

1 Answer 1

6
$\begingroup$

You can see how this sort of thing can happen by looking at an extreme case. Sorry I can't show you SAS code, but here is a dummy example in R:

library(randomForest)
library(MASS)
N <- 1000 #1000 observations
p <- 10 #10 variables
X <- mvrnorm(N, rep(0, p), diag(rep(1, p))) #The variables are all normal and independent
y <- X[,1] + rnorm(N, sd = .1) #the outcome is a function of the first of the 10 variables plus random noise

using all the variables, looking at 1/3rd of them by default at each split:

rf1 <- randomForest(y = y, x = X)

using only the good variable:

rf2 <- randomForest(y = y, x = X[,1, drop = FALSE])

using all the variables, but tying all the variables at each split

rf3 <- randomForest(y = y, x = X, mtry = 10)

1> rf1

Call:
 randomForest(x = X, y = y) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 3

          Mean of squared residuals: 0.04463831
                    % Var explained: 95.32
1> rf2

Call:
 randomForest(x = X[, 1, drop = FALSE], y = y) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 1

          Mean of squared residuals: 0.01377959
                    % Var explained: 98.56
1> rf3

Call:
 randomForest(x = X, y = y, mtry = 10) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 10

          Mean of squared residuals: 0.01192359
                    % Var explained: 98.75

So you see that random forest is not completely invulnerable to noise variables. Why does this happen? Because in each split of each tree, the method of random subspaces will select from among the available variables and the useful ones may not be among that split!

When we set the algo to look at each variable at each split, we get the same performance as when we only look at the good one.

$\endgroup$
7
  • $\begingroup$ I wish I could give this answer more than one up vote. This is the best, short explanation, I think I have ever seen. $\endgroup$ Jan 26, 2017 at 20:43
  • $\begingroup$ Thank you for this great explanation! My only remaining question is this. Let's say you run a random forest only using your good variable, so it's just bootstrapping the rows and running 500 trees on those randomly sampled sets with one variable. You now try to validate this model and it performs worse than taking the good variable with no model. What I mean is that if I simply rank my score on the validation set from low to high and put it into 5 bins, THAT outperforms all of my random forest models on the validation set. This part I don't get. The score with no model is beating all models. $\endgroup$
    – stats217
    Jan 26, 2017 at 20:56
  • 1
    $\begingroup$ When you say "no model" you mean "a simple categorical model" right? It probably has to do with the nature of the score and the categories. $\endgroup$ Jan 26, 2017 at 21:04
  • $\begingroup$ What I mean is that the score is a discrete number, let's say it ranges from 0-100. If I bin this into 5 equally populated groups and find the bad rate, that outperforms all models. This itself is not a model, just a rank ordering of a score. When I use this score in a model it performs worse than the score itself. Does that make sense? $\endgroup$
    – stats217
    Jan 26, 2017 at 21:08
  • 1
    $\begingroup$ well if the true model mapping the score to the outcome is approximately linear, then a random forest will pick this up -- but not nearly as efficiently as a model that bakes in linearity structurally. RF is basically a nonparametric regression, and as such will always underperform a correctly specified parametric model in finite samples $\endgroup$ Jan 26, 2017 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.