# XGBoost Objective Derivation Problem

This is the loss function of XGBoost.

This is the Second-order approximation of the loss function.

Note:

$$$$L^{(t)} \text{: cross entropy loss function.}$$$$

$$$$y_i \text{: label. Marked as y, as a matter of convenience.}$$$$

$$$$\hat{y_i}^{(t-1)} \text{: probability of previous tree. Marked as p, as a matter of convenience.}$$$$

$$$$f_t(x_i) \text{: output value of the tree }$$$$

$$$$g_i = \frac{ ∂l(y, p) }{∂p} = \frac{p-y}{p(p-1)}$$$$

$$$$h_i = \frac{ ∂l(y, p) }{∂^2p} = \frac{\left(y-p\right)p+y\left(p-1\right)}{\left(p-1\right)^2p^2}$$$$

Why does everyone in the gi and hi calculation do :

$$$$g_i = \frac{ ∂l(y, p) }{∂x} = p-y$$$$

$$$$h_i = \frac{ ∂l(y, p) }{∂^2x} = p(1-p)$$$$

NOTE: the x is:

$$$$x = ln(\frac{p}{1 - p})$$$$

and the yi hat (t-1) is sigmoid function:

$$$$p = \frac{e^x}{1+e^x} = \frac{1}{1+e^{-x}}$$$$

Why are the derivatives are with respect to x (or y_hat in the code) instead of p? $$$$g_i = \frac{ ∂l(y, p)}{∂p} => \frac{ ∂l(y, p) }{∂x}$$$$

Since you mention probabilities, I assume you are thinking about a binary classification, in which case the trees all operate in the log-odds space, the $$x$$s in your notation. So the $$\hat{y}$$ and $$f_t$$ are actually log-odds not probabilities.
• Thanks for your reply and help. Now that XGBoost paper says: $$gi=\frac{∂l(y,p)}{∂p}$$, how can we change p to x: $$gi=\frac{∂l(y,p)}{∂x}$$ It's totally not the same equation. May 10, 2022 at 8:52