2
$\begingroup$

I'm trying to learn how to calibrate random forests for regression. I have found many informative sources on how to do this for classification but none for regression. The CORElearn R package has a random forest classifier with local weighting of basic models. But nothing for regression. I am basically trying to correct the bias random forests present when attacking a regression problem. A problem that I have seen worries many people but I have not seen explicit solutions to yet. Any new thoughts on this?

$\endgroup$
1
  • $\begingroup$ What does calibation even mean for regression? Calibration, as I understand it, is a problem unique to classification when the estimated class probabilities are not aligned (that is, not calibrated) with the proportions of actual incidences of the estimated class. I do not think this problem is relevant to regression. $\endgroup$
    – Tripartio
    Commented Dec 9, 2022 at 7:24

1 Answer 1

4
$\begingroup$

I believe the bias in RFs is "a feature, not a bug." Decision trees have a tendency to overfit the data, so the bias in random forests counter-acts the overfit somewhat, making a random forest model more robust than a straight-forward decision tree otherwise would have been. If you want to further attack overfit, you may be interested in using regularized trees for feature selection: http://arxiv.org/pdf/1201.1587.pdf, also check out the RRF package.

Of course, this doesn't answer your question which was what options are available for tuning your model. For one automated approach, the randomForest package has a tuneRF() function for tuning the mtry paramter (the number of variables sampled in each tree).

$\endgroup$
3
  • $\begingroup$ Whether decision trees overfit depends on their depth, doesn't it? $\endgroup$
    – Nameless
    Commented Jul 9, 2013 at 14:47
  • $\begingroup$ I definetly think that it is a feature. The ensemble produced in random forests through bagging + Ho's random subspace selection is a variance reduction method. Meaning the descrease in MSE is achieved through diminishing variance but keeping the bias present in any of the trees of the ensemble. Even so, when trying to solve regression tasks the bias is very annoying because you lose the capability to predict the "high values" and "low values" of your sample correctly. $\endgroup$
    – JEquihua
    Commented Jul 9, 2013 at 14:49
  • $\begingroup$ Random forests are designed to not over fit, but it might still happen for some datasets and you can overcorrelate your ensmble if you have too many trees. $\endgroup$
    – JEquihua
    Commented Jul 9, 2013 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.