How to compute $g_i$ and $h_i$, i.e. the first and second derivative of the loss function in XGBoost?

Question

In XGBoost, the objective function is $J(f_i)=\sum_{i=1}^{n}L(y_i,\hat{y}_i^{(t-1)}+f_t(x_i))+\Omega{(f_i)}+C$, If we take Taylor expansion of the objective function and let $$g_i=\frac{\partial{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$, and $$h_i=\frac{\partial^2{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$ If the loss function $L$ is the loss function of logistic regression, i.e. $L=-\sum_{i=1}^{m}y_ilog(h_i)+(1-y_i)log(1-h_i)$,then I think $$g_i=\frac{\partial{L}}{\partial{h_i}}=-(\frac{y_i}{h_i}+(1-y_i)*\frac{-1}{1-h_i})$$,that is $$g_i=-\frac{y_i-h_i}{h_i(1-h_i)}$$ However, in the example given by the XGBoost package, they think $g_i$ of loss function of logistic regression is $g_i=h_i-y_i$. Here is the g and h definition:

def logregobj(preds, dtrain):
    labels = dtrain.get_label()
    preds = 1.0 / (1.0 + np.exp(-preds))
    grad = preds - labels
    hess = preds * (1.0-preds)
    return grad, hess

full code can be found here. I don't get it. Can anyone help? Thanks in advance!

Well, it seems that there is another version which I think is correct.

def custom_loss(y_pre,D_label): 
    label=D_label.get_label()
    penalty=2.0
    grad=-label/y_pre+penalty*(1-label)/(1-y_pre) 
    hess=label/(y_pre**2)+penalty*(1-label)/(1-y_pre)**2 
    return grad,hess

Although the penalty seems wired.

Answer 1

I think that part of the misunderstanding stems from using the symbol $h$ in two different places for two different meanings.

The code portion of the question seems to have little to do with the mathematics of XGBoost, since the code snippets are not part of the XGBoost software. Denote the binary cross-entropy loss for a single sample $$ L(y_i, \hat{y}_i) = -\left[ y_i \log(\hat{y}_i) + (1- y_i) \log(1 - \hat{y}_i) \right]. $$ The loss for the model is $\sum_i L(y_i, \hat{y}_i)$. This is a quantity that we want to minimize.

The authors provide that $g_i = \partial_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right)$, with the notation $\text{something}^{(t-1)}$ denoting that this is the prediction for trees up to and including tree number $t-1$. We can write, dropping indices on $y$ because life is short,

$$ \begin{align} g_i &= \frac{\partial L}{\partial \hat{y}} L(y, \hat{y}) \\ &=\frac{y}{\hat{y}} - \frac{1 - y}{1 - \hat{y}} \\ &=\frac{y(1 - \hat{y}) - \hat{y}(1-y)}{\hat{y}(1 - \hat{y})} \\ &= \frac{ y - y\hat{y} - \hat{y}+y\hat{y} }{\hat{y}(1 - \hat{y})} \\ &= \frac{y - \hat{y}}{\hat{y}(1 - \hat{y})} \end{align} $$

For $h_i$, we can follow the same procedure.

$$ \begin{align} h_i &= \partial^2_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right) \\ &= \frac{\partial}{\partial \hat{y}} g_i \\ &= \frac{\partial}{\partial \hat{y}} \left[\frac{y - \hat{y}}{\hat{y}(1 - \hat{y})}\right] \\ &= \frac{ y - 1}{(\hat{y} -1)^2} - \frac{y}{\hat{y}^2} \end{align} $$

but remember that for compactness/ease of reading I dropped all of the super- and sub-scripts.

Answer 2

The discrepancy is due to the interpretation of $y_i^{t-1}$. In your derivation, you're assuming it is the probability $h_i$, whereas the code author has defined it as the log odds $logit(h_i) = log(\frac{h_i}{1-h_i})$. Re-express the loss as a function of log odds instead of probability (define $O_i = logit(h_i)$):

$$ L_i = -y_i O_i + log(1 + exp(O_i)) $$

And find the derivative with respect to the log odds:

$$ g_i = \frac{d L_i}{d O_i} = h_i - y_i $$

(Side note: As stated by @Sycorax, you're overloading the term $h_i$ because the xgboost paper authors define it as the 2nd order gradient statistic)

Answer 3

As mentioned by @StayLearning, in slide 4, the author defines the logistic loss $L = \sum_{i=1}^n l(y_i,\hat y_i)$ where $$ l(y_i,\hat y_i) = y_i\log(1+\exp(-\hat y_i)) + (1-y_i)\log(1+\exp(\hat y_i)) $$ then grad =

\begin{align} \frac{\partial l}{\partial \hat y_i} &= - y_i\times\frac{1}{1+\exp(\hat y_i)} + (1-y_i)\times\frac{\exp(\hat y_i)}{1+\exp(\hat y_i)}\\ &= \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} - y_i \end{align}

and hess = $$ \frac{\partial^2 l}{(\partial \hat y_i)^2} = \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} \times \frac{1}{1+\exp(\hat y_i)}, $$

where preds = 1.0 / (1.0 + np.exp(-yhat_i)) and label = y_i.

See more here.