16
$\begingroup$

I am exploring different classification methods for a project I am working on, and am interested in trying Random Forests. I am trying to educate myself as I go along, and would appreciate any help provided by the CV community.

I have split my data into training/test sets. From experimentation with random forests in R (using the randomForest package), I have been having trouble with a high misclassification rate for my smaller class. I have read this paper concerning the performance of random forests on imbalanced data, and the authors presented two methods with dealing with class imbalance when using random forests.

1. Weighted Random Forests

2. Balanced Random Forests

The R package does not allow weighting of the classes (from the R help forums, I have read the classwt parameter is not performing properly and is scheduled as a future bug fix), so I am left with option 2. I am able to specify the number of objects sampled from each class for each iteration of the random forest.

I feel uneasy about setting equal sample sizes for random forests, as I feel like I would be losing too much information about the larger class leading to poor performance with future data. The misclassification rates when downsampling the larger class has shown to improve, but I was wondering if there were other ways to deal with imbalanced class sizes in random forests?

$\endgroup$
  • $\begingroup$ Am also facing the similar problem. Could you please provide the script for Weighted Random Forest and Metric based random forest, if you have?? Looking forward for your reply $\endgroup$ – user6414 Sep 21 '11 at 10:39
4
$\begingroup$

If you don't like those options, have you considered using a boosting method instead? Given an appropriate loss function, boosting automatically recalibrates the weights as it goes along. If the stochastic nature of random forests appeals to you, stochastic gradient boosting builds that in as well.

$\endgroup$
  • $\begingroup$ Hi David, thanks for the suggestion. Would boosting be possible within/in combination with random forests? Also, I have read that boosting may not be applicable with noisy data. The data that I am working with contains many "noisy" variables- I am currently running random forests without feature selection, downsampling the larger class to deal with imbalance. I have been getting acceptable results (correctly classifying 85% of the smaller class and 93% of the larger class in the training set), but improvements are always welcome. $\endgroup$ – ialm Jun 7 '11 at 16:18
  • $\begingroup$ @Jonathan I don't have a good sense of how much noise is required before random forests start outcompeting the various boosting methods, but my impression is that it's quire a lot. The gbm package in R has a functioning "weights" term, and you can increase the randomness in a very similar fashion to random forests by decreasing with the "bag.fraction" term. It seems like it's worth a try. $\endgroup$ – David J. Harris Jun 7 '11 at 20:41
  • $\begingroup$ @DavidJ.Harris Boosting also suffers from class imbalance, especially in the case of absolute rarity, when information about the minority classes is sparse $\endgroup$ – Antoine Apr 26 '15 at 19:53
2
$\begingroup$

I think that weighting objects is somehow equivalent to duplicating them. Maybe you should try modifying the bootstrap step by sampling appropriately your different classes.

Another thought is that class imbalance may shift your decision threshold to another value than $0.5$ (if it's a binary classification problem). Try considering ROC curves and AUC to evaluate how bad the imbalance is causing poor performances on your models.

$\endgroup$
0
$\begingroup$

Instead of sampling large classes you can expand small classes ! If large classes have many times more observation then small, then biase will be small. I do hope you can handle that supersized dataset.

You may also identify subsets of observations which handle the most information about large classes, there are many possible procedures, the simplest I think is based on nearest neighbors method - observation sampling conditioned on neighborhood graph structure guarantee that sample will have probability density more similar to original one.

randomForest is written in Fortran and c, source code is available (http://cran.r-project.org/src/contrib/randomForest_4.6-2.tar.gz) but I cant spot the place where enthropy is computed, ps. ups that randomforest use Gini instead of enthropy

$\endgroup$
  • $\begingroup$ Identifying subsets of observations in the larger class seems like an interesting idea. The second part of the project is actually differentiating the larger class into smaller, useful subgroups that may be useful for future study. As for the entropy calculations in the code, I have not actually delved into the code and have only looked at some of the theory behind rf, so I cannot comment on that. $\endgroup$ – ialm Jun 7 '11 at 16:21
  • $\begingroup$ I wanted to modify that code long time ago, but lacked time for that, adding weights shoudnt be conceptually difficult. $\endgroup$ – Qbik Jun 7 '11 at 23:02
  • $\begingroup$ Random forest in R could be use as regression tool, mayby it's useful for unsupervised classification too.I wanted to modify code of randomForest long time ago, but lacked time for that, adding weights shoudnt be conceptually difficult. Time ago I was wondering on my own how to use random forest for unsupervised clusterisation and I got an idea but didnt tested that yet, unfortunatly it seems rather computionally demanding - numerofobservations^2. $\endgroup$ – Qbik Jun 7 '11 at 23:09
0
$\begingroup$

(1) You're right the weighting function doesn't work and not sure if it has ever been fixed.

(2) Most use option 2 with balanced data. The key to not loosing too much data is stratified sampling. You randomly sample a unique balanced set for each tree.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.