3
$\begingroup$

i have been wondering for some time now how random forests (or AdaBoost, doesn't matter) are built when using cross-validation. Let's see we're using 5-fold cross validation to train random forests on 5 different training sets and therefore test on 5 different test sets. How does the 'final' random forest look like when we are basically building 5 random forests (one for each fold of the cross validation). How are these forests combined into a final model?

I have never understood this step and I really hope someone can help me with this!

thanks in advance, Steven

$\endgroup$
1
  • 2
    $\begingroup$ You train it on the training data to get a final model. Cross-validation is just to make decisions about the parameters/models etc. $\endgroup$
    – Michael M
    Commented Jul 25, 2020 at 10:46

2 Answers 2

1
$\begingroup$

As @vivek says, random forests often don't need cross-validation, though you can use cross-validation to choose the 'number of predictors per node' tuning parameter. Even there, you'd probably just use the out-of-bag prediction error that random-forest implementations typically give you for free rather than cross-validation.

Boosted trees tend to have more tuning parameters that are worth adjusting (especially XGBoost, which has ALL THE PARAMETERS). The basic approach is the same as cross-validation for a cost-complexity tuning parameter in regression or lasso or CART. You pick a set of values for the tuning parameter, and for each value, use cross-validation to estimate your favorite out-of-sample prediction error metric (eg, mean squared prediction error). You then choose the tuning-parameter value that minimises this metric and re-run the boosting algorithm with that chosen value.

If you want to tune multiple parameters this can be slow: some people use a 'greedy' algorithm where they tune one parameter then fix it and tune another parameter, then fix that and tune a third parameter, and so on. This will work ok as long as the best value for one parameter isn't too sensitive to the values of other parameters.

In principle, the sort of experimental designs used for response-surface optimisation should get you better results, but I haven't seen people doing it very much.

$\endgroup$
0
$\begingroup$

I am not sure why you are using cross-validation with RandomForest. RandomForest does not need cross validation. When you train a RF model, each tree uses bootstrapped samples from original data as train set and leaves about 1/3 of data called out of bag(oob) data. Each oob data not used in training is marked and then used for validation using the forest(data is tested on forest not on a tree). Out of bag data is used for each tree to take vote on and finally we average them to get the final result.

Random Forest does not need cross-validation to avoid overfitting. It uses (bootstrapping + averaging) called as bagging to deal with overfitting.

$\endgroup$
1
  • $\begingroup$ That has not been my experience. When I bootstrap the entire process to assess absolute prediction accuracy (i.e., calibration-in-the-small) I have seen some of the worst overfitting of any method I have tried. I see a lot of practitioners only validating things like classification accuracy; they would be shocked at the miscalibration of RFs in my experience. Thomas what do you think? $\endgroup$ Commented Mar 9, 2023 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.