35
$\begingroup$

I got a an R script from someone to run a random forest model. I modified and run it with some employee data. We are trying to predict voluntary separations.

Here is some additional info: this is a classification model were 0 = employee stayed, 1= employee terminated, we are currently only looking at a dozen predictor variables, the data is "unbalanced" in that the term'd records make up about 7% of the total record set.

I run the model with various mtry and ntree selections but settled on the below. The OOB is 6.8% which I think is good but the confusion matrix seems to tell a different story for predicting terms since the error rate is quite high at 92.79% Am I right in assuming that I can't rely on and use this model because the high error rate for predicting terms? or is there something also I can do to use RF and get a smaller error rate for predicting terms?

 FOREST_model <- randomForest(theFormula, data=trainset, mtry=3, ntree=500, importance=TRUE, do.trace=100)
ntree      OOB      1      2
  100:   6.97%  0.47% 92.79%
  200:   6.87%  0.36% 92.79%
  300:   6.82%  0.33% 92.55%
  400:   6.80%  0.29% 92.79%
  500:   6.80%  0.29% 92.79%
> print(FOREST_model)

Call:
 randomForest(formula = theFormula, data = trainset, mtry = 3,      ntree = 500, importance = TRUE, do.trace = 100) 
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 3

        OOB estimate of  error rate: 6.8%
Confusion matrix:
     0  1 class.error
0 5476 16 0.002913328
1  386 30 0.927884615
> nrow(trainset)
[1] 5908
$\endgroup$
20
$\begingroup$

The confusion matrix is calculated at a specific point determined by the cutoff on the votes. Depending on your needs, i.e., better precision (reduce false positives) or better sensitivity (reduce false negatives) you may prefer a different cutoff.

For this purpose I recommend plotting (i) a ROC curve, (ii) a recall-precision and (iii) a calibrating curve in order to select the cutoff that best fits your purposes. All these can be easily plotted using the 2 following functions from the ROCR R library (available also on CRAN):

pred.obj <- prediction(predictions, labels,...)
performance(pred.obj, measure, ...)

For example:

rf <- randomForest (x,y,...);
OOB.votes <- predict (rf,x,type="prob");
OOB.pred <- OOB.votes[,2];

pred.obj <- prediction (OOB.pred,y);

RP.perf <- performance(pred.obj, "rec","prec");
plot (RP.perf);

ROC.perf <- performance(pred.obj, "fpr","tpr");
plot (ROC.perf);

plot  (RP.perf@alpha.values[[1]],RP.perf@x.values[[1]]);
lines (RP.perf@alpha.values[[1]],RP.perf@y.values[[1]]);
lines (ROC.perf@alpha.values[[1]],ROC.perf@x.values[[1]]);
$\endgroup$
  • $\begingroup$ I am interested in your answer, I plotted with the code you shared. will you please give me some resources to find a bit detail about the plot you suggested. or will write few sentences about how to interpret it. many thanks in advance. $\endgroup$ – MKS Jul 8 '16 at 12:33
  • $\begingroup$ I suggest that you start with the entry for ROC curve that linked to above and other entries mentioned there. $\endgroup$ – Itamar Jul 9 '16 at 17:26
  • $\begingroup$ The answer doesn't directly address the limitation of decision tree in a highly skewed data set. $\endgroup$ – SmallChess Oct 27 '16 at 4:28
19
$\begingroup$

Your set is sharply unbalanced -- RF usually fails in this scenario (i.e. predicts well only the bigger class).

You should try balancing your set either by sampling the "0" class only to have about the same size as "1" class or by playing with classwt parameter.

$\endgroup$
  • 1
    $\begingroup$ Despite there being a classwt parameter, I don't think it is implemented yet in the randomForest() function of the randomForest package in R. Per Link. Have you used it before? I tried it with different values but got identical results to the default classwt=NULL. $\endgroup$ – Zhubarb Sep 23 '15 at 7:38
9
$\begingroup$

Based on your confusion matrix, you've got 5,908 data points and the vast, vast majority of them are of type 0 ('employee stayed'). The classifier can therefore get away with being "lazy" and picking the majority class unless it's absolutely certain that an example belongs to the other class. Note that your overall error rate is ~7%, which is quite close to the percent of Class1 examples!

You've got a few options:

  • Discard Class0 examples until you have roughly balanced classes. I don't know if there's literature on how to choose an optimally representative subset (maybe someone else can weigh in?), but you could start by dropping examples at random. You can pass a subset argument to randomForest, which should make this trivial to test.

  • Adjust your loss function/class weights to compensate for the disproportionate number of Class0. You essentially want to make it much more expensive for the classifier to misclassify a Class1 example than Class0 one. It might make sense to try Class0 = 1/0.07 ~= 14x Class1 to start, but you may want to adjust this based on your business demands (how much worse is one kind of error). I think the classwt parameter is what you're looking for here.

  • Use stratified sampling to ensure that you've got examples from both classes in the trees' training data. It's possible that some of your trees were trained on only Class0 data, which will obviously bode poorly for their generalization performance. Check out the strata argument.

$\endgroup$
  • $\begingroup$ Randomly selecting from the dominant class sounds reasonable. They don't need to be equal: even a 1:5 ratio should be an improvement. $\endgroup$ – Itamar Jun 20 '12 at 11:35
  • $\begingroup$ @Itmar,that's definitely what I would try first. However, it seems like there must be some way to ensure that the examples you retain are representative of the larger data set. $\endgroup$ – Matt Krause Jun 28 '12 at 1:01
  • 1
    $\begingroup$ One option could be to run PCA (principle component analysis) on the larger set, cluster the points according to the first principle component and pick one sample from each cluster. $\endgroup$ – Itamar Jun 28 '12 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.