5
$\begingroup$

Suppose I am using random forests where the classes are highly unbalanced. How do you detect over fitting and what can you do to avoid it? Breiman says in his paper that random forests do not overfit, but others say that they can? If overfitting does exist(i.e. correlated trees) what is the best course of action to counteract that, and why does Breiman say that Random Forests are impervious to overfitting?

Also, how do you deal with the fact that the class you want to predict is percent-wise so small(suppose you have 99% 1's and 1% 0's)? What are some key metrics to measure the overall model fit and how do you go about testing and training in such cases?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ One way you could analyze overfitting is by plotting learning curves. $\endgroup$ – JNevens Apr 18 '15 at 13:53
  • $\begingroup$ Any updates on this? Are random forests impervious to overfitting? $\endgroup$ – lord12 Apr 19 '15 at 18:04
6
$\begingroup$

In highly unbalanced datasets, how do you detect over fitting?

Use metrics that are robust against unbalanced datasets, like Precision, Recall or F1-score. In your example with 99% 1's and 1% 0's. A classifier that always predicts positive samples will an Accuracy of 0.99, but Precision, Recall or F1-score of 0.00

What can you do to avoid overfitting in unbalanced datasets?

  • Cluster you positive samples into several clusters of the same size of the negative samples, i.e. move from binary classification to multi-label classification
  • Sub-sample the positive samples in order to have a 50/50 dataset.
  • Use other algorithm that deals naturally with unbalanced datasets, like anomaly detection methods.

Do Random Forest overfit?

Yes. Any classifier with high complexity respect to the training data will overfit. However, the overfitting will not increase when the number of single Decision Trees is increased.

How to avoid overfitting with Random Forest?

  • Decrease the complexity of the Decision Tree: pre- or post-pruning
  • Randomly drop features and/or samples per node
Improve this answer
$\endgroup$
5
  • $\begingroup$ If one really (really really) have to make RF more robust, pruning is not best way to go. Instead reduce bootstrap sample size and grow some more trees. $\endgroup$ – Soren Havelund Welling Nov 18 '15 at 17:49
  • $\begingroup$ ...random sub-sampling(down sampling) is fine. But works best if performed individually during bootstrap of each tree. Otherwise are some samples never used. $\endgroup$ – Soren Havelund Welling Nov 18 '15 at 17:51
  • $\begingroup$ Pruning will help to decrease the overfitting as it limits the complexity (variance) of the model $\endgroup$ – tashuhka Nov 19 '15 at 9:49
  • $\begingroup$ True, but reducing bootstrap sample size also reduce tree complexity and one get more tree decorrelation and faster training as bonus. $\endgroup$ – Soren Havelund Welling Nov 19 '15 at 16:22
  • $\begingroup$ This answer is missing the most important way to deal with imbalanced classes: use probabilities instead of assigning classes! $\endgroup$ – Matthew Drury Jun 13 '18 at 21:03
4
$\begingroup$

I personally think Random Forests are not impervious to overfitting. Overfitting is always a possibility, for any model. One possible way to counteract overfitting is by always using cross-validation.

If you want to detect overfitting, you can plot learning curves. Here, you are going to train the model multiple times, each time with a larger training set. Afterwards, you calculate the score on both the training set and the test set and plot these scores. Or, as Scikit Learn puts it: "A learning curve shows the validation and training error of an estimator for varying numbers of training samples." If the training error is low and the validation error is much higher, your model is overfitting.

To address your second question; when your data consists of 99% 1's and 1% 0's, this will certainly affect your final result!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.