2
$\begingroup$

I might not fully understand the topic but: Bootstrapped/bagged trees are difficult to interpret because the decisions are made from averaging the prediction of possibly hundreds of trees (ensemble).

But with boosted trees, don't we end up with one tree?
If so, visualizing this tree should be as simple as visualizing a normal decision tree, right?
The only difference would be that it's more unclear how it got to those specific rules since it uses boosting.

However, I've also read that it's difficult to interpret a boosted tree. Could someone please clarify?

Improve this question
$\endgroup$
2
6
$\begingroup$

In short, the final ensemble is not a tree.

Consider this part of the Wikipedia article.

The $F_m(x)$ when $m=1$ is essentially a single tree $h_1(x)$ with its output multiplied by a learned number $\gamma_m$.

In the next iteration, you build $F_2(x)$ by using the previously created $F_1(x)$ and another initial tree $h_2(x)$.

This operation is repeated $m$ times, and then, for example if there were $m=100$ initial trees, you get your final ensemble, $F_{100}(x)=F_{99}(x)+F_{98}(x)+...+F_{1}(x) + \gamma_{100}h_{100}(x)$.

When you visualize a tree, you see the decision nodes. The output of the initial tree follows from its nodes.
However, the output of the final ensemble follows from a hundred of trees giving their outputs, then multiplying these outputs by the learned values and adding it all together.

Therefore, the final ensemble is not a tree, so you cannot visualize it as a tree. A tree is a decision graph, when a learned ensemble is a linear combination of what many trees decide to yield for a given input.

(Thanks to usεr11852 for clarifying my error. The comment section is about the previous version of the post before the fix.)

Improve this answer
$\endgroup$
6
  • $\begingroup$ I think that the "in short" sentence is misleading and dangerous for novice users - it would be wrong to think that there is "one (hell of a) tree"; boosted trees are an archetypical example of ensemble learners and that sentence misrepresents that. The rest of the answer is fine. $\endgroup$ – usεr11852 May 7 '20 at 12:24
  • $\begingroup$ @usεr11852, thank you so much for the critique! It will help me to understand the theme better. I read that you said "The rest of the answer is fine". In this "rest" of the answer, I named F100 as the "final tree". Is my naming convention for F100 wrong? I understand that F100 is an ensemble created from previous iterations of the function F. Could you please elaborate on why it is wrong to call F100 a tree and what is the correct naming convention? $\endgroup$ – Dmitrii Badretdinov May 8 '20 at 7:43
  • 1
    $\begingroup$ Sure thing. 0. (Sorry, I misread your post a bit.) I think you mean $h_{100}(x) = y - (h_{99}(x) + h_{98}(x) + \dots + h_{1}(x))$ in the sense GBM are additive models.1. Each output $F_i$ from a tree $h_i$ models the deviance residuals in the sense $F_{i} = F_{i-1} + a_i h_i$ where $F$ minimises our expected loss. 2. Because it is an ensemble of trees (as you correctly state) there is no single tree representation more than we would have a single representation for a random forest or neural network or saying a GBM is actually a linear model with tens of thousands of step functions. (cont.) $\endgroup$ – usεr11852 May 8 '20 at 9:30
  • $\begingroup$ 3. $F_{100}$ is therefore not "the final tree" but rather the "final prediction" of 100 trees. ($a_i$ is our step size.) $\endgroup$ – usεr11852 May 8 '20 at 9:30
  • 3
    $\begingroup$ +1. This is good. Thank you for taking the time to amend it. $\endgroup$ – usεr11852 May 18 '20 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.