1
$\begingroup$

This question already has an answer here:

Using the following R code I obtain a decision tree using the agaricus dataset:

data(agaricus.train, package='xgboost')

bst <- xgboost(data = agaricus.train$data, label = agaricus.train$label, max_depth = 3,
               eta = 1, nthread = 2, nrounds = 2,objective = "binary:logistic")
# plot all the trees
xgb.plot.tree(model = bst)
# plot only the first tree and display the node ID:
xgb.plot.tree(model = bst, trees = 0, show_node_id = TRUE)

I want to understand more clearly the "value" output of the tree (the 3rd line in the oval shaped object). Here we can see that tree 0 leaf 7 gives a value 1.90174532. (That is the first terminal node in the image). I want to know if this value is the same as the log-odds score. So, all observations which follow the upper path of the decision tree will obtain a log-odds score of 1.90174532. Then in a new decision tree the observations will fall into a different split depending on each observations characteristics and will obtain a "new" value Then we sum up all these values across all trees to obtain a final log-odds score which can then be converted to a predicted probability using the logistic function.

Is my intuition correct? Does value = log-odds.

( https://rdrr.io/cran/xgboost/man/xgb.plot.tree.html )

enter image description here

$\endgroup$

marked as duplicate by gung r Dec 1 '18 at 2:04

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ I think you will find the information you need in the linked thread. Please read it. If it isn't what you want / you still have a question afterwards, come back here & edit your question to state what you learned & what you still need to know. Then we can provide the information you need without just duplicating material elsewhere that already didn't help you. $\endgroup$ – gung Dec 1 '18 at 2:05
  • $\begingroup$ Yes, thank you for the link. I understand it, I just have one additional question. $\endgroup$ – user113156 Dec 5 '18 at 17:21
  • $\begingroup$ The answer states 'Each node in the tree has an associated "weight."' So using the tree example in my original question each decision node will have a "weight" which is not displayed. That is node 0-0 will have a weight, node 0-2 will have another weight, node 0-5 will have an additional weight. Then the terminal node is a summation over all the "weights" preceding it, in nodes 0-0, 0-2 and 0-5 which depending on the side of the split you obtain a value (or final weight) of 0.8085 or -1.985. Is this correct? Finally..... $\endgroup$ – user113156 Dec 5 '18 at 17:29
  • $\begingroup$ XGBoost constructs another tree (tree 1) where the features are in different positions in the tree and the split numbers are also different. Then the observations will fall into different terminal nodes and because the features are in different positions in the tree the summation of all the "weights" in the terminal node will be different. So an observation which fell into terminal node 0-12 in tree 0 may now fall into terminal node 1-8 in tree 1. With two terminal scores/weights. This will go on for say 100 trees in our model, so each observation will have 100 terminal weight-scores.......... $\endgroup$ – user113156 Dec 5 '18 at 17:35
  • $\begingroup$ These weights/scores are then summed up and we obtain a log-odds score for each observation which can be converted to a predicted probability. - Is this correct? $\endgroup$ – user113156 Dec 5 '18 at 17:36
2
$\begingroup$

The "value" is the contribution of a leaf to the logit. The logit for a sample is the sum of the "value" of all of a sample's leafs. Because XGBoost is an ensemble, a sample will terminate in one leaf for each tree; gradient boosted ensembles sum over the predictions of all trees.

Then the logit can be used in the ordinary way, such as computing the predicted probability of class membership.

More information about gradient boosted trees generally and XGBoost specifically can be found in https://arxiv.org/abs/1603.02754 or .

$\endgroup$
  • $\begingroup$ Thanks for your comment, its a little more clearer now, under my original post I have made some additional comments. Is my intuition correct? $\endgroup$ – user113156 Dec 5 '18 at 20:30
  • 1
    $\begingroup$ @user113156 It sounds like you have some additional questions. You can ask a new question by clicking the "ask question" button. $\endgroup$ – Sycorax Dec 5 '18 at 20:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.