0
$\begingroup$

I would like to understand XGBoost in detail. In Figure 2 of the paper, 5 instances are displayed together with their gradient and their hessian.

Assuming I want to find the optimal split in the current tree, I need to compute the gradient and the hessian of each observation. 1) Based on the assumption of a squared loss function, how can I do this?

As far as I understand, Gradient Boosting fits the next base learner (regression tree in my case) to the pseudo-residuals. These can be calculated as a result from the most current boosting estimator at step m, so f_{m-1}(x).

Now, in case of XGBoost I do not only need the gradient, which I think are the pseudo residuals, but also the Hessian. So what is the hessian in this case? Is it the sign of the pseudo-residuals?

2) I tried to visualise my computations. Especially, I tried to use the Scoring function provided in the original paper in order to evaluate a possible split. I cannot figure out, what is wrong with them. The first split results in a loss of zero, which seems not to be ok, since it does not reflect a perfect fit. (Do I really have to evaluate the gradients at each x_i first before taking their sum?)

As far as I understood, I have to apply the scoring function (equation (7)) in the original paper for every possible split. In my case, I assume, that there are no trees grown yet, which means, that I am in iteration m=1. I assume, that in m=0, the mean value in the target is the initial guess.

My example was supposed to show the evaluation of the first possible split, namely splitting the first variable in the first split possible. I know, that I can operate two further splits for X1 and three splits for X2, so 6 splits in total and I would have to evaluate all of them with the Scoring function.

However, how can I evaluate the first possible split of the first possible split variable with the scoring function? I know, that the scoring function yield the loss reduction, so I would calculate the Scoring function for every possible split and split the tree at the variable and split point maximising the Score. I guess, that my calculation is not correct yet, so I would really appreciate help!

Thanks in advance!

Initial guess

Evaluate split 1 of variable X1

Edit: This is the paper

Improve this question
$\endgroup$
1
  • $\begingroup$ Could you provide a link to the paper your mention? It's also not at all clear which calculations are yours, and which are from the figure. $\endgroup$
    – Alex R.
    Commented Feb 21, 2018 at 18:10

1 Answer 1

Reset to default
1
$\begingroup$

I think you're getting confused. In the paper XGBoost: A Scalable Tree Boosting System, the calculations you are referring to are to calculate the optimal weight and whether to make a split in the first place. The gradient and hessian here refer to the first and second derivatives of the loss function. Explicitely you are summing over all trees thus far, and all samples. In practice you would likely just use a batch of samples.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Good answer. (+1) I think the easiest way to understand what XGBoost is doing when using the Hessian is to try to have an analogy between gradient descent and Newton's method. In contrast with other libraries (eg. gbm in R) where custom cost functions require simply the first derivative, XGBoost explicitly requires the second derivative too. $\endgroup$
    – usεr11852
    Commented Feb 20, 2018 at 22:15
  • $\begingroup$ Ok, I am still struggling and edited my post. Can somebody help? $\endgroup$
    – Sylvi0202
    Commented Feb 21, 2018 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.