12
$\begingroup$

I have a question purely theoretical about decision trees outputs for classification. I have heard a lot of people say "the output of tree models are not probabilities", and having studied those I don't understand why they would say that.

My reasons to say that they are in fact probabilities are:

1. Simple Classification Tree: Each leaf in a decision tree represent a subset of the training data, with a certain distribution of classes. The proportion of each class in a leaf can be interpreted as a probability estimate for belonging to that class. For instance, if a leaf contains 80 samples of Class A and 20 samples of Class B, the probability estimate for Class A would be 80% and for Class B would be 20%.

2. Ensemble Models (Random Forests): In a Random Forest, multiple decision trees are trained on different subsets of the data and/or features. The predictions of these trees are then aggregated. For classification, this typically involves taking a majority vote or averaging the predicted probabilities from each tree. By averaging the predicted probabilities across many trees, the Random Forest can provide a more reliable probability estimate than a single decision tree. This further supports the view that tree-based models can output probabilities.

3. Boosting Models (Gradient Boosting, XGBoost): Boosting models like Gradient Boosting or XGBoost sequentially build trees, with each tree correcting errors made by the previous ones. For binary classification, the final output is often in the form of log-odds, which then gets transformed to probabilities using a logistic function (sigmoid function). This transformation explicitly produces probability estimates.

Reasons to say that they are not probabilities are:

1. Calibration: It is known that many classification problems deal with unbalanced dataset, for this, many people uses parameters such as is_unbalance or class_weights to compensate for this. Using these parameters make the output not be well-calibrated. This means that the predicted probability does not necessarily correspond to the true likelihood of the event. For example, a leaf might predict a class with 80% probability, but in reality, the true likelihood of that class might be different. Calibration techniques, such as Platt scaling or isotonic regression, are often applied to improve this. Although, I believe that this reason is valid, using these parameters is just an special case of tree-models so I wouldn't be so fast to say tree-based models do not produce probabilities.

2. Overfitting: We all agree that decision trees can easily overfit (although I would debate that a bit more in the case of Random Forest), especially those that are not pruned, leading to overconfident probability estimates. This overfitting can result in leaves with very few samples, which makes the probability estimates less reliable. I agree somewhat with this statement, however, this is simply telling me that it is a poorly trained model, it does not mean that essentially tree-based models do not output probabilities, it is simply the case of a poorly trained model like any other. Same situation could happen for a poorly trained Neural Networks or even a logistic regression.

I would love to hear your thoughts and opinions about the issue and learn something new.

$\endgroup$
1
  • $\begingroup$ As you pointed out with calibration, usually the actual probabilities don't matter for classifiers, as long as the classification is robust. $\endgroup$
    – qwr
    Commented Jun 23 at 1:00

2 Answers 2

10
$\begingroup$

Your assessment of the situation is excellent. I would just add that in my practice random forests suffer some of the worst miscalibration that I’ve ever witnessed as a statistician.

Even a single tree method like recursive partitioning (CART) can output probabilities. This has been done for 40 years. I wouldn’t trust them as they use small denominators in the proportions computed at the terminal nodes, and thus are imprecise from not borrowing information across nodes.

$\endgroup$
2
$\begingroup$

I think this can be made simpler. Single independent variable, and classification. In this case, I would argue, tree-based model is not that dissimilar from learning the distribution of your binary variable as a function of the independent variable. More specifically learning two distributions - of targets and of non-targets. It does seem that this could, in principle work. In practice, you will hit problems with some regions of your empirical distribution not having any targets at all. I would argue that this same problem will occur in tree based models, but at a much more significant scale.

[Personal Observation]: In practice, I don't really see the need for having probability. I usually use classifiers to execute some sort of a policy: 'obervations with score>X get selected, others get dropped'. I then measure the implications of that policy as a whole, and that is what matters. I may want probability distributions for the key metrics. Whether score is probability or not - I don't think I care much, as long as there is a way to split targets from non-targets

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.