# How to compute $g_i$ and $h_i$, i.e. the first and second derivative of the loss function in XGBoost?

In XGBoost, the objective function is $J(f_i)=\sum_{i=1}^{n}L(y_i,\hat{y}_i^{(t-1)}+f_t(x_i))+\Omega{(f_i)}+C$, If we take Taylor expansion of the objective function and let $$g_i=\frac{\partial{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$, and $$h_i=\frac{\partial^2{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$ If the loss function $L$ is the loss function of logistic regression, i.e. $L=-\sum_{i=1}^{m}y_ilog(h_i)+(1-y_i)log(1-h_i)$,then I think $$g_i=\frac{\partial{L}}{\partial{h_i}}=-(\frac{y_i}{h_i}+(1-y_i)*\frac{-1}{1-h_i})$$,that is $$g_i=-\frac{y_i-h_i}{h_i(1-h_i)}$$ However, in the example given by the XGBoost package, they think $g_i$ of loss function of logistic regression is $g_i=h_i-y_i$. Here is the g and h definition:

def logregobj(preds, dtrain):
labels = dtrain.get_label()
preds = 1.0 / (1.0 + np.exp(-preds))
hess = preds * (1.0-preds)


full code can be found here. I don't get it. Can anyone help? Thanks in advance!

Well, it seems that there is another version which I think is correct.

def custom_loss(y_pre,D_label):
label=D_label.get_label()
penalty=2.0
hess=label/(y_pre**2)+penalty*(1-label)/(1-y_pre)**2


Although the penalty seems wired.

• The code you've linked to isn't the official XGBoost code, it's someone's code to demonstrate how to specify custom loss functions. I think that explains at least part of the discrepancy.
– Sycorax
Jul 5, 2018 at 19:48
• The reason that there's a peculiar penalty scalar included in the second code block is because the author is using it as an example of a customized loss function.
– Sycorax
Jul 6, 2018 at 1:32

I think that part of the misunderstanding stems from using the symbol $h$ in two different places for two different meanings.

The code portion of the question seems to have little to do with the mathematics of XGBoost, since the code snippets are not part of the XGBoost software. Denote the binary cross-entropy loss for a single sample $$L(y_i, \hat{y}_i) = -\left[ y_i \log(\hat{y}_i) + (1- y_i) \log(1 - \hat{y}_i) \right].$$ The loss for the model is $\sum_i L(y_i, \hat{y}_i)$. This is a quantity that we want to minimize.

The authors provide that $g_i = \partial_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right)$, with the notation $\text{something}^{(t-1)}$ denoting that this is the prediction for trees up to and including tree number $t-1$. We can write, dropping indices on $y$ because life is short,

\begin{align} g_i &= \frac{\partial L}{\partial \hat{y}} L(y, \hat{y}) \\ &=\frac{y}{\hat{y}} - \frac{1 - y}{1 - \hat{y}} \\ &=\frac{y(1 - \hat{y}) - \hat{y}(1-y)}{\hat{y}(1 - \hat{y})} \\ &= \frac{ y - y\hat{y} - \hat{y}+y\hat{y} }{\hat{y}(1 - \hat{y})} \\ &= \frac{y - \hat{y}}{\hat{y}(1 - \hat{y})} \end{align}

For $h_i$, we can follow the same procedure.

\begin{align} h_i &= \partial^2_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right) \\ &= \frac{\partial}{\partial \hat{y}} g_i \\ &= \frac{\partial}{\partial \hat{y}} \left[\frac{y - \hat{y}}{\hat{y}(1 - \hat{y})}\right] \\ &= \frac{ y - 1}{(\hat{y} -1)^2} - \frac{y}{\hat{y}^2} \end{align}

but remember that for compactness/ease of reading I dropped all of the super- and sub-scripts.

• homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf. The author define y_hat different from how you are using it (slide 3). Using the author's slide, you can easily get g = sigma(y_hat) and h = g*(1-g) Jul 22, 2019 at 21:44
• @StayLearning You should add that as an answer! Click the "Answer Question" button to write an answer showing how to apply the materials in the slide to answer this question.
– Sycorax
Jul 22, 2019 at 22:40

As mentioned by @StayLearning, in slide 4, the author defines the logistic loss $$L = \sum_{i=1}^n l(y_i,\hat y_i)$$ where $$l(y_i,\hat y_i) = y_i\log(1+\exp(-\hat y_i)) + (1-y_i)\log(1+\exp(\hat y_i))$$ then grad =

\begin{align} \frac{\partial l}{\partial \hat y_i} &= - y_i\times\frac{1}{1+\exp(\hat y_i)} + (1-y_i)\times\frac{\exp(\hat y_i)}{1+\exp(\hat y_i)}\\ &= \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} - y_i \end{align}

and hess = $$\frac{\partial^2 l}{(\partial \hat y_i)^2} = \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} \times \frac{1}{1+\exp(\hat y_i)},$$

where preds = 1.0 / (1.0 + np.exp(-yhat_i)) and label = y_i.

See more here.