How to interpret OOB and confusion matrix for random forest?

Question

I got a an R script from someone to run a random forest model. I modified and run it with some employee data. We are trying to predict voluntary separations.

Here is some additional info: this is a classification model were 0 = employee stayed, 1= employee terminated, we are currently only looking at a dozen predictor variables, the data is "unbalanced" in that the term'd records make up about 7% of the total record set.

I run the model with various mtry and ntree selections but settled on the below. The OOB is 6.8% which I think is good but the confusion matrix seems to tell a different story for predicting terms since the error rate is quite high at 92.79% Am I right in assuming that I can't rely on and use this model because the high error rate for predicting terms? or is there something also I can do to use RF and get a smaller error rate for predicting terms?

 FOREST_model <- randomForest(theFormula, data=trainset, mtry=3, ntree=500, importance=TRUE, do.trace=100)
ntree      OOB      1      2
  100:   6.97%  0.47% 92.79%
  200:   6.87%  0.36% 92.79%
  300:   6.82%  0.33% 92.55%
  400:   6.80%  0.29% 92.79%
  500:   6.80%  0.29% 92.79%
> print(FOREST_model)

Call:
 randomForest(formula = theFormula, data = trainset, mtry = 3,      ntree = 500, importance = TRUE, do.trace = 100) 
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 3

        OOB estimate of  error rate: 6.8%
Confusion matrix:
     0  1 class.error
0 5476 16 0.002913328
1  386 30 0.927884615
> nrow(trainset)
[1] 5908

Answer 1

The confusion matrix is calculated at a specific point determined by the cutoff on the votes. Depending on your needs, i.e., better precision (reduce false positives) or better sensitivity (reduce false negatives) you may prefer a different cutoff.

For this purpose I recommend plotting (i) a ROC curve, (ii) a recall-precision and (iii) a calibrating curve in order to select the cutoff that best fits your purposes. All these can be easily plotted using the 2 following functions from the ROCR R library (available also on CRAN):

pred.obj <- prediction(predictions, labels,...)
performance(pred.obj, measure, ...)

For example:

rf <- randomForest (x,y,...);
OOB.votes <- predict (rf,x,type="prob");
OOB.pred <- OOB.votes[,2];

pred.obj <- prediction (OOB.pred,y);

RP.perf <- performance(pred.obj, "rec","prec");
plot (RP.perf);

ROC.perf <- performance(pred.obj, "fpr","tpr");
plot (ROC.perf);

plot  ([email protected][[1]],[email protected][[1]]);
lines ([email protected][[1]],[email protected][[1]]);
lines ([email protected][[1]],[email protected][[1]]);

Answer 2

Your set is sharply unbalanced -- RF usually fails in this scenario (i.e. predicts well only the bigger class).

You should try balancing your set either by sampling the "0" class only to have about the same size as "1" class or by playing with classwt parameter.

Answer 3

Based on your confusion matrix, you've got 5,908 data points and the vast, vast majority of them are of type 0 ('employee stayed'). The classifier can therefore get away with being "lazy" and picking the majority class unless it's absolutely certain that an example belongs to the other class. Note that your overall error rate is ~7%, which is quite close to the percent of Class1 examples!

You've got a few options:

• Discard Class0 examples until you have roughly balanced classes. I don't know if there's literature on how to choose an optimally representative subset (maybe someone else can weigh in?), but you could start by dropping examples at random. You can pass a subset argument to randomForest, which should make this trivial to test.

• Adjust your loss function/class weights to compensate for the disproportionate number of Class0. You essentially want to make it much more expensive for the classifier to misclassify a Class1 example than Class0 one. It might make sense to try Class0 = 1/0.07 ~= 14x Class1 to start, but you may want to adjust this based on your business demands (how much worse is one kind of error). I think the classwt parameter is what you're looking for here.

• Use stratified sampling to ensure that you've got examples from both classes in the trees' training data. It's possible that some of your trees were trained on only Class0 data, which will obviously bode poorly for their generalization performance. Check out the strata argument.