6
$\begingroup$

I used a two-step cforest in my model. the accuracy of the train set is 87%, yet the accuracy of the test set is 57%. This indicates the model is severely overfitting. How to solve this problem? Should I reduce the nodes of the tree or divide the data into k fold? How can I determine how many nodes should I retain?

Here is the code for step 1.

   fit1 <- cforest((b == 'three')~ posemo + social + family
            +friend + home + humans + money + they
            + social+article+certain+insight+affect+ negemo+ future+swear+sad
            +negate+ppron+sexual+death + filler+leisure, data = trainset1, 
            controls=cforest_unbiased(ntree=3000, mtry= 3))
Improve this question
$\endgroup$

3 Answers 3

Reset to default
15
$\begingroup$

In random forests, overfitting is generally caused by over growing the trees

as stated in one of the other answers is completely WRONG. The RF algorithm, by definition, requires fully grown unprunned trees. This is the case because RF can only reduce variance, not bias (where $error=bias+variance$). Since the bias of the entire forest is roughly equal to the bias of a single tree, the base model used has to be a very deep tree to guarantee a low bias. Variance is subsequently reduced by growing many deep, uncorrelated trees and averaging their predictions.

I wouldn't necessarily say that a training accuracy of 87% and a test accuracy of 57% indicates severe overfitting. Performance on your training set will always be higher than on your test set. Now, you need to provide more information if you want CV users to be able to diagnose the source of your potential overfitting problem.

  • how did you tune the parameters of your random forest model? Did you use cross-validation, or an independent test set? What are the sizes of your training/testing sets? Did you properly used randomization to constitute these sets?
  • is your target categorical or continuous? If yes to the former, do you have any kind of class imbalance issue?
  • how did you measure error? If it applies, is your classification problem binary, or multiclass?

In practice, Random Forest seldom overfit. But what would tend to favor overfitting would be having too many trees in the forest. At some point it is not necessary to keep adding trees (it does not reduce variance anymore, but can slightly increase it). This is why the optimal number of trees should be optimized like any other hyperparameter (or at least, should not be carelessly set to too high of a number. It should be the smallest number of trees needed to achieve lowest error. You can look at a plateau in the curve of OOB error VS number of trees).

Other than overfitting, the difference in accuracy between train & test that you observe could be explained by differences between the sets. Are the same concepts present in both sets? If not, even the best classifier won't be able to perform well out-of-bag. You can't extrapolate for something if you did not even learn about some aspect of it.

I would also recommend that you read the section about RF in the formative Elements of Statistical Learning. Especially, see section 15.3.4 (p. 596) about RF and overfitting.

Improve this answer
$\endgroup$
9
  • 4
    $\begingroup$ +1, but I think it's misleading to state that ntree has to be optimized like any other parameter, which might suggest a grid over mtry and ntree. Because trees are i.i.d., it suffices to generate a large number of trees conditional on other hyperparameters and then to check out-of-bag error rates as a function of ntree, picking a small value of ntree after the error rate stabilizes. $\endgroup$
    – Sycorax ♦
    Sep 14, 2015 at 18:00
  • 1
    $\begingroup$ I actually always use a grid search to optimize ntree and mtry... $\endgroup$
    – Antoine
    Sep 14, 2015 at 18:10
  • $\begingroup$ I agree that picking the smallest value of ntree leading to stabilization of the error is the way to go, but if you do it at the end once the other parameters have been chosen, how can you be sure that another combination of parameters would not have returned an even lower error plateau? $\endgroup$
    – Antoine
    Sep 14, 2015 at 18:20
  • 2
    $\begingroup$ Conditional on selecting other hyperparameters, RF trees are i.i.d. Fixing other hyperparameters and estimating RFs for ntree={500,1000,1500} is the same as fixing hyperparameters and estimating an RF for ntree=1500 and then checking how well the first 500, first 1000 and first 1500 perform. $\endgroup$
    – Sycorax ♦
    Sep 14, 2015 at 18:45
  • 1
    $\begingroup$ Suppose your boss asks you to flip a coin 50 times and record H or T. (Flips are i.i.d.) You do so. Then he asks you to instead increase the sample size to 100. What's the minimum number of additional flips you need to do to have a sample size representing 100 i.i.d. flips? $\endgroup$
    – Sycorax ♦
    Sep 14, 2015 at 19:11
4
$\begingroup$

One reason that you Random Forest may be overfitting may be because you have a lot of redundant features or your features are heavily correlated.

If lot of your features are redundant, then when you perform the splits in the nodes of the trees, the algorithm may often only choose very poor features, which makes your model just chase some noise in your data.

It is wise to look at the variable importance of the forest to try to identify if you have features that are maybe not relevant.

You can also try to make some dimensionality-reduction/aggregation on the features.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ +1 nice answer. Somewhat related: When the number of variables is large, but the fraction of relevant variables small, random forests are likely to perform poorly with small m. At each split the chance can be small that the relevant variables will be selected. and When the number of relevant variables increases, the performance of random forests is surprisingly robust to an increase in the number of noise variables. For example, with 6 relevant and 100 noise variables, the probability of a relevant variable being selected at any split is 0.46 Both from my link above $\endgroup$
    – Antoine
    Sep 14, 2015 at 18:00
  • 1
    $\begingroup$ Exactly! I think this is one of the main reason people people get "bad" performance with RFs. They just dump in all the features without doing any prior analysis. $\endgroup$
    – Gumeo
    Sep 14, 2015 at 18:05
-5
$\begingroup$

In random forests, overfitting is generally caused by over growing the trees.

Pruning the trees would also help.

So, some parameters which you can optimize in the cForest argument are the ntree, mtry

mtry is the number of variables the algorithm draws to build each tree. ntree is the total number of trees in the forest.

Having said that, cross validation always helps. Consider carrying out k-fold cross validation. This forum on Kaggle would help understand carrying out cross validation in random forests.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ prune single tree models, not random forest trees $\endgroup$
    – Soren Havelund Welling
    Aug 30, 2015 at 22:54
  • $\begingroup$ @Lucia please consider unaccepting the current accepted answer as it is misleading $\endgroup$
    – Antoine
    Sep 14, 2015 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.