3
$\begingroup$

I understand that the probability in the random forest algorithm uses the count of trees that vote for a certain class, while logistic regression uses MLE. What are the implications of using one vs another? And is the RF probability a true probability?

$\endgroup$
  • 2
    $\begingroup$ Most random forest implementations average the probabilities predicted from the individual trees, instead of forcing the trees to hard vote and then counting the hard votes. $\endgroup$ – Matthew Drury Mar 28 '18 at 13:49
4
$\begingroup$

In a nutshell, logistic regression aims to produce an estimation of the probability of belonging to a specific class. So there is only one "probability estimate" after a logistic regression. On the other hand, the probability obtained using random forest is more like a by product, taking advantage of having many trees (though this is implementation dependent! more details below) and therefore, there are many ways to infer probabilities from a random forest.

Random forest probability

Indeed, it is not a true probability, in the sense that it is just an average over the number of trees.

For the implication, they will depend on the penalty function that you use. Usually, random forest will produce many ties (in terms of probabilities) and 0 and 1. This is not good when your metric is the AUC (see this article on wikipedia if you are not familiar with AUC), because of the ties, and not good either when you observe a logarithmic loss (because the 0 and 1 can have a large impact on the penalty).

However, there as some alternatives to improve the estimation of probabilities, as detailed here.

H. Boström. Estimating class probabilities in random forests. In Proc. of the International Conference on Machine Learning and Applications, pages 211–216, 2007.

Logistic regression probability

Usually, they produce a good estimate of the probability. But as opposed to random forest, they do not take into account possible interactions of the input. So it may harm performance as well. I suspect that in most cases, if you penalty is just the accuracy of the model (and some interactions are important) a logistic regression would give poor results compared to a random forest.

$\endgroup$
  • 3
    $\begingroup$ You can always add interactions to logistic regression model $\endgroup$ – seanv507 Mar 29 '18 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.