2
$\begingroup$

I have a dataset that I'm trying to classify into 2 groups, A and B, using a random forest model. I know the true grouping and I'm trying to see how well I can model it using the other available variables. I've tried 2 different approaches that I thought would be equivalent, but which are actually giving me quite different results:

  1. Reading in the grouping as a (non-numeric) factor in R, growing a classification forest, and taking the proportion of trees that vote for group A as my prediction.
  2. Constructing an indicator variable for membership of group A, growing a regression forest, and taking the ensemble prediction as usual.

The split between the 2 groups is roughly 90-10 A vs. B. I'm growing 240 trees from ~200k observations of the same variables. I've left most of the settings at the defaults for the R randomForest package, but to keep the processing time down to a manageable level I've increased the node size to 200. The results are as follows:

  1. In the vast majority of cases, all 240 trees vote for A. The average predicted chance of any one observation being in A is about 99.9%. Worse still, not a single member of group B gets a majority of votes for group B!
  2. I get a wide range of predictions, with the mean prediction lying close to the observed mean of ~90%.

How can two apparently similar methods give such different results?

As for how I ended up trying this - I was initially trying to classify my dataset into a larger number of groups, of which B was one, but I noticed that B was being classified almost 100% incorrectly. The other groups are all much better behaved, even though most of them make up a far smaller proportion of my data.

$\endgroup$
  • 3
    $\begingroup$ how do you construct the indicator variable for membership of group A? Why would you think these approaches are smiliar? On the other hand, you have a highly unbalanced problem (90% A and 10% B if I understood correctly) since the default setting of random forests is simply to take the majority vote that's why A always wins. There are much more A observations. You can try balancing the model by doing randomForest(..., sampsize=c(size,size),...) where size has to be small than the size of your smaller class. Maybe like 60% of it since the objective of random forest is to introduce variation. $\endgroup$ – JEquihua May 29 '13 at 3:35
  • $\begingroup$ I'd have expected on average about 90% of the trees in the classification forest to vote for A, but what I'm actually seeing is very close to 100% every time. I'm interested in the % of trees voting for each group, not just which one has a majority of votes. $\endgroup$ – user3490 May 29 '13 at 7:26
  • $\begingroup$ I think it could either be beacause you are using the default setting so it's taking 0.632 of A and 0.632 of B for each tree it builds. But each tree is not grown till there is only one observation in each node so the proportion is less favorable than a tree built on the whole data set. It could also be because you have only 240 trees, random forests are like nearest neighbour mechanisms, so if you dont have enough trees the random sampling might not grab your nearest. Another thing could simply be that your clases are not separable. If you do a balanced forest is the classification good? $\endgroup$ – JEquihua May 29 '13 at 16:41
  • $\begingroup$ How many training examples do you have? It is important. $\endgroup$ – user31264 Oct 28 '13 at 18:17
4
$\begingroup$

Due to the class imbalance, you should have a look at the probabilities that your forests outputs (I'm not familiar with the random forest R package, but I think there is an option (type="prob") in the predict function that will give you a matrix of class probabilities.

I believe, the next thing to do with these probabilities is to derive a ROC curve and see if it performs better than the majority vote. In that case, it just means you should consider a 'soft' voting approach while optimising the threshold (based on the ROC curve) to determine the predicted class (which is straightforward in a binary case) instead of a 'majority' voting one.

$\endgroup$
  • $\begingroup$ Yes it is likely because of class imbalance. You can also look into various tweaks to random forest for class imbalanced data including using class weights ("Weighted Random Forest") or sampling the data to balance the classes ("Roughly Balanced Bagging"). $\endgroup$ – Ryan Bressler Mar 31 '14 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.