I have been playing around with post processing the results of the random forest for regression machine learning algorithm in order to try and do better than the default mean of all trees prediction. Chapter 16 of Elements of statistical learning talks about how and why this can be done using regularised regression (in that case elastic net).

On the other hand, I routinely re-calibrate the results of random forests using a logistic model, since I often see the kind of bias discussed in this question, and find that a simple linear logistic regression model tends to account for it. I have found that the calibration needs to be done on the ensemble prediction, attempting to calibrate individual trees doesn't work nearly as well.

I am unsure then of how to proceed when you want to do both of these things. If you do the elastic net regression first, you are finding the optimal weighted mean/subset of the forest ensemble for the biased prediction. I see no reason to be sure that this will be the optimal combination for a re-calibrated prediction. I don't see how the elastic net analysis could be run simultaneously with re-calibration, as the calibration is non-linear.

The non-linear problem of mutuality optimising the calibration parameters as well as weights and sub-set selection is doable is principle, but seem almost intractable computationally for large data sets.

Can anyone suggest an efficient path for achieving both of these aims in an optimised way?

  • 1
    $\begingroup$ Sorry, I dont know how to answer your question buy I am quite interested on what you mention on re-calibrating the results of random forests using a logistic model. Could you give me some pointers on that subject? Thank you. $\endgroup$ – JEquihua Feb 20 '13 at 19:46
  • 1
    $\begingroup$ See the question linked in the second paragraph of the question above. It shows a nice example of the kind of bias in RF predictions that I've often seen, and some good answers. $\endgroup$ – Bogdanovist Feb 20 '13 at 22:39
  • 2
    $\begingroup$ See also the discussion here kaggle.com/c/bioresponse/forums/t/1974/rf-and-boost-tree. There is a link to a nice paper on this at in post #4. $\endgroup$ – Bogdanovist Feb 20 '13 at 22:41
  • $\begingroup$ Thank you! By the way Have you had any luck post-processing the results of random forest regression to do better than the mean of the sequence of trees? I looked for the "elastic net" approach in the Elements of statistical... in the chapter on random forests but did not find it. Could you detail this a bit more or share any experience you have had? I'm very very interest in what you describe. Greeetings and thank you. $\endgroup$ – JEquihua Feb 22 '13 at 0:23
  • 3
    $\begingroup$ Sad to not be able to get the rest of the conversation. $\endgroup$ – EngrStudent Apr 21 '17 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.