1
$\begingroup$

I am running a random forest for different sets of data, with an attempt to make it dynamic enough to optimize for all sets of data (they are are similar data sets). There are around 150 predictor variables. What is the main reason for underfitting in a random forest? How do you achieve the optimal amount of bias and variance. Which one is better to introduce over the other? Better yet is, there an intuitive way to make an RF optimize using the parameters - "n"-number of trees grown and "r" to solve for this. I moved away from OLS to see if RF could handle a much larger amount of predictor variables with a smaller set of sample data.

Any help is much appreciated!

Improve this question
$\endgroup$
2
  • $\begingroup$ RF can handle more predictor variables, but your question appears to be too general. Please read previous discussions under this tag and specify your question. $\endgroup$ – katya May 12 '15 at 18:34
  • $\begingroup$ Is this question about how to do hyperparameter optimization of random forests? $\endgroup$ – Michael M Jul 6 '17 at 6:44
1
$\begingroup$

The traditional method (AFAIK) to choose hyperparameters such as the n=number of trees in the forest, d=depth of each tree, c=node splitting criterion etc is cross validation.

eg : Set a small portion (20% for starters) of your training data set aside, train the RF on the remaining 80% and evaluate its performance on the 20% that you set aside earlier. Choose the n that gives you best performance on the 20% set. (This is a simple example, you might want to use something like k fold cross validation.)

Improve this answer
$\endgroup$
0
$\begingroup$

Each predictor variable costs you one degree of freedom. When your variables equal your observations, then you have a saturated model. Even before you reach saturation, the cost of numerous variables can destroy on your model's strength. Then there is the multicollinearity that occurs when you use a lot of variables.

Improve this answer
$\endgroup$
1
  • 6
    $\begingroup$ This doesn't seem relevant to the question. OP is talking about random forests, not linear models, and concepts like degrees of freedom don't translate neatly. $\endgroup$ – Hong Ooi Jul 9 '15 at 22:13
0
$\begingroup$

People generally don't worry about underfit, unless they can only train the model in a computer 20 years ago ( which takes forever to train a model complex enough due to computational speed). You can increase the number of trees and the number of splits each tree can have ( generally by generating a search grid ) and do a cross validated training on each parameter combination, and see how your prediction accuracy changes with the change of tree number and complexity of the single tree. At the point of the accuracy stops improving, that's the point when you definitely do not underfit, and more likely overfit.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.