How to compute $g_i$ and $h_i$, i.e. the first and second derivative of the loss function in XGBoost?

In XGBoost, the objective function is $J(f_i)=\sum_{i=1}^{n}L(y_i,\hat{y}_i^{(t-1)}+f_t(x_i))+\Omega{(f_i)}+C$, If we take Taylor expansion of the objective function and let $$g_i=\frac{\partial{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$, and $$h_i=\frac{\partial^2{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$ If the loss function $L$ is the loss function of logistic regression, i.e. $L=-\sum_{i=1}^{m}y_ilog(h_i)+(1-y_i)log(1-h_i)$,then I think $$g_i=\frac{\partial{L}}{\partial{h_i}}=-(\frac{y_i}{h_i}+(1-y_i)*\frac{-1}{1-h_i})$$,that is $$g_i=-\frac{y_i-h_i}{h_i(1-h_i)}$$ However, in the example given by the XGBoost package, they think $g_i$ of loss function of logistic regression is $g_i=h_i-y_i$. Here is the g and h definition:

def logregobj(preds, dtrain):
labels = dtrain.get_label()
preds = 1.0 / (1.0 + np.exp(-preds))
hess = preds * (1.0-preds)


full code can be found here. I don't get it. Can anyone help? Thanks in advance!

Well, it seems that there is another version which I think is correct.

def custom_loss(y_pre,D_label):
label=D_label.get_label()
penalty=2.0
hess=label/(y_pre**2)+penalty*(1-label)/(1-y_pre)**2


Although the penalty seems wired.

• The code you've linked to isn't the official XGBoost code, it's someone's code to demonstrate how to specify custom loss functions. I think that explains at least part of the discrepancy.
– Sycorax
Commented Jul 5, 2018 at 19:48
• The reason that there's a peculiar penalty scalar included in the second code block is because the author is using it as an example of a customized loss function.
– Sycorax
Commented Jul 6, 2018 at 1:32

I think that part of the misunderstanding stems from using the symbol $h$ in two different places for two different meanings.

The code portion of the question seems to have little to do with the mathematics of XGBoost, since the code snippets are not part of the XGBoost software. Denote the binary cross-entropy loss for a single sample $$L(y_i, \hat{y}_i) = -\left[ y_i \log(\hat{y}_i) + (1- y_i) \log(1 - \hat{y}_i) \right].$$ The loss for the model is $\sum_i L(y_i, \hat{y}_i)$. This is a quantity that we want to minimize.

The authors provide that $g_i = \partial_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right)$, with the notation $\text{something}^{(t-1)}$ denoting that this is the prediction for trees up to and including tree number $t-1$. We can write, dropping indices on $y$ because life is short,

\begin{align} g_i &= \frac{\partial L}{\partial \hat{y}} L(y, \hat{y}) \\ &=\frac{y}{\hat{y}} - \frac{1 - y}{1 - \hat{y}} \\ &=\frac{y(1 - \hat{y}) - \hat{y}(1-y)}{\hat{y}(1 - \hat{y})} \\ &= \frac{ y - y\hat{y} - \hat{y}+y\hat{y} }{\hat{y}(1 - \hat{y})} \\ &= \frac{y - \hat{y}}{\hat{y}(1 - \hat{y})} \end{align}

For $h_i$, we can follow the same procedure.

\begin{align} h_i &= \partial^2_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right) \\ &= \frac{\partial}{\partial \hat{y}} g_i \\ &= \frac{\partial}{\partial \hat{y}} \left[\frac{y - \hat{y}}{\hat{y}(1 - \hat{y})}\right] \\ &= \frac{ y - 1}{(\hat{y} -1)^2} - \frac{y}{\hat{y}^2} \end{align}

but remember that for compactness/ease of reading I dropped all of the super- and sub-scripts.

• homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf. The author define y_hat different from how you are using it (slide 3). Using the author's slide, you can easily get g = sigma(y_hat) and h = g*(1-g) Commented Jul 22, 2019 at 21:44
• @StayLearning You should add that as an answer! Click the "Answer Question" button to write an answer showing how to apply the materials in the slide to answer this question.
– Sycorax
Commented Jul 22, 2019 at 22:40

The discrepancy is due to the interpretation of $$y_i^{t-1}$$. In your derivation, you're assuming it is the probability $$h_i$$, whereas the code author has defined it as the log odds $$logit(h_i) = log(\frac{h_i}{1-h_i})$$. Re-express the loss as a function of log odds instead of probability (define $$O_i = logit(h_i)$$):

$$L_i = -y_i O_i + log(1 + exp(O_i))$$

And find the derivative with respect to the log odds:

$$g_i = \frac{d L_i}{d O_i} = h_i - y_i$$

(Side note: As stated by @Sycorax, you're overloading the term $$h_i$$ because the xgboost paper authors define it as the 2nd order gradient statistic)

As mentioned by @StayLearning, in slide 4, the author defines the logistic loss $$L = \sum_{i=1}^n l(y_i,\hat y_i)$$ where $$l(y_i,\hat y_i) = y_i\log(1+\exp(-\hat y_i)) + (1-y_i)\log(1+\exp(\hat y_i))$$ then grad =

\begin{align} \frac{\partial l}{\partial \hat y_i} &= - y_i\times\frac{1}{1+\exp(\hat y_i)} + (1-y_i)\times\frac{\exp(\hat y_i)}{1+\exp(\hat y_i)}\\ &= \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} - y_i \end{align}

and hess = $$\frac{\partial^2 l}{(\partial \hat y_i)^2} = \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} \times \frac{1}{1+\exp(\hat y_i)},$$

where preds = 1.0 / (1.0 + np.exp(-yhat_i)) and label = y_i.

See more here.