3
$\begingroup$

As far as I know, for classification problems, there's generally a lack of a measure of probabilities in machine learning models, unless a customized probability structure is designed. Random forest, however, has a unique way of estimating probabilities, by counting the number of times a specific class is voted by trees, which I think is a byproduct of the model.

Here, my questions are how good these probabilities are and how much I can trust these probabilities, in places like betting, where the probabilities play an important role. Thanks.

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ What is a good probability? $\endgroup$ – Sycorax says Reinstate Monica Jan 27 '17 at 15:20
  • $\begingroup$ Are you asking about whether these probabilities are well calibrated? Here is an example and some discussion on calibration (or lack thereof) of probabilities obtained from Random Forest. $\endgroup$ – darXider Jan 27 '17 at 15:34
  • $\begingroup$ @Sycorax If we could rerun the experiment infinite times, the outcomes have an observed frequency close to the actual probability. That's the definition of probability, right? i.e., if the world described by random forest is the real world where the data generating process happens. $\endgroup$ – Maxareo Jan 27 '17 at 15:39
  • $\begingroup$ @darXider My question is if the probabilities are close the truth. Is calibration helping getting the probabilities closer to the truth? $\endgroup$ – Maxareo Jan 27 '17 at 15:41
  • 1
    $\begingroup$ @Maxareo That's not the definition of a probability. A probability just has to satisfy the Kolmogorov axioms. You're asking about whether the RF probabilities match the true class membership, i.e. calibration. $\endgroup$ – Sycorax says Reinstate Monica Jan 27 '17 at 15:58
3
$\begingroup$

They aren't all that well-calibrated, but they're not terrible. What you're looking for is Platt scaling. Basically, take the scores from the RF and fit a logitic regression to them.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.