In XGBoost, the objective function is $J(f_i)=\sum_{i=1}^{n}L(y_i,\hat{y}_i^{(t-1)}+f_t(x_i))+\Omega{(f_i)}+C$, If we take Taylor expansion of the objective function and let $$g_i=\frac{\partial{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$, and $$h_i=\frac{\partial^2{L(y_i,\hat{y}_i^{(t-1)})}}{\partial{\hat{y}_i^{(t-1)}}}$$ If the loss function $L$ is the loss function of logistic regression, i.e. $L=-\sum_{i=1}^{m}y_ilog(h_i)+(1-y_i)log(1-h_i)$,then I think $$g_i=\frac{\partial{L}}{\partial{h_i}}=-(\frac{y_i}{h_i}+(1-y_i)*\frac{-1}{1-h_i})$$,that is $$g_i=-\frac{y_i-h_i}{h_i(1-h_i)}$$ However, in the example given by the XGBoost package, they think $g_i$ of loss function of logistic regression is $g_i=h_i-y_i$. Here is the g and h definition:

def logregobj(preds, dtrain):
    labels = dtrain.get_label()
    preds = 1.0 / (1.0 + np.exp(-preds))
    grad = preds - labels
    hess = preds * (1.0-preds)
    return grad, hess

full code can be found here. I don't get it. Can anyone help? Thanks in advance!

Well, it seems that there is another version which I think is correct.

def custom_loss(y_pre,D_label): 
    return grad,hess

Although the penalty seems wired.

  • $\begingroup$ The code you've linked to isn't the official XGBoost code, it's someone's code to demonstrate how to specify custom loss functions. I think that explains at least part of the discrepancy. $\endgroup$ – Sycorax Jul 5 '18 at 19:48
  • $\begingroup$ The reason that there's a peculiar penalty scalar included in the second code block is because the author is using it as an example of a customized loss function. $\endgroup$ – Sycorax Jul 6 '18 at 1:32

I think that part of the misunderstanding stems from using the symbol $h$ in two different places for two different meanings.

The code portion of the question seems to have little to do with the mathematics of XGBoost, since the code snippets are not part of the XGBoost software. Denote the binary cross-entropy loss for a single sample $$ L(y_i, \hat{y}_i) = -\left[ y_i \log(\hat{y}_i) + (1- y_i) \log(1 - \hat{y}_i) \right]. $$ The loss for the model is $\sum_i L(y_i, \hat{y}_i)$. This is a quantity that we want to minimize.

The authors provide that $g_i = \partial_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right)$, with the notation $\text{something}^{(t-1)}$ denoting that this is the prediction for trees up to and including tree number $t-1$. We can write, dropping indices on $y$ because life is short,

$$ \begin{align} g_i &= \frac{\partial L}{\partial \hat{y}} L(y, \hat{y}) \\ &=\frac{y}{\hat{y}} - \frac{1 - y}{1 - \hat{y}} \\ &=\frac{y(1 - \hat{y}) - \hat{y}(1-y)}{\hat{y}(1 - \hat{y})} \\ &= \frac{ y - y\hat{y} - \hat{y}+y\hat{y} }{\hat{y}(1 - \hat{y})} \\ &= \frac{y - \hat{y}}{\hat{y}(1 - \hat{y})} \end{align} $$

For $h_i$, we can follow the same procedure.

$$ \begin{align} h_i &= \partial^2_{\hat{y}_i^{(t-1)}} L\left(y_i, \hat{y}_i^{(t-1)}\right) \\ &= \frac{\partial}{\partial \hat{y}} g_i \\ &= \frac{\partial}{\partial \hat{y}} \left[\frac{y - \hat{y}}{\hat{y}(1 - \hat{y})}\right] \\ &= \frac{ y - 1}{(\hat{y} -1)^2} - \frac{y}{\hat{y}^2} \end{align} $$

but remember that for compactness/ease of reading I dropped all of the super- and sub-scripts.

  • $\begingroup$ homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf. The author define y_hat different from how you are using it (slide 3). Using the author's slide, you can easily get g = sigma(y_hat) and h = g*(1-g) $\endgroup$ – StayLearning Jul 22 '19 at 21:44
  • $\begingroup$ @StayLearning You should add that as an answer! Click the "Answer Question" button to write an answer showing how to apply the materials in the slide to answer this question. $\endgroup$ – Sycorax Jul 22 '19 at 22:40

As mentioned by @StayLearning, in slide 4, the author defines the logistic loss $L = \sum_{i=1}^n l(y_i,\hat y_i)$ where $$ l(y_i,\hat y_i) = y_i\log(1+\exp(-\hat y_i)) + (1-y_i)\log(1+\exp(\hat y_i)) $$ then grad =

\begin{align} \frac{\partial l}{\partial \hat y_i} &= - y_i\times\frac{1}{1+\exp(\hat y_i)} + (1-y_i)\times\frac{\exp(\hat y_i)}{1+\exp(\hat y_i)}\\ &= \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} - y_i \end{align}

and hess = $$ \frac{\partial^2 l}{(\partial \hat y_i)^2} = \frac{\exp(\hat y_i)}{1+\exp(\hat y_i)} \times \frac{1}{1+\exp(\hat y_i)}, $$

where preds = 1.0 / (1.0 + np.exp(-yhat_i)) and label = y_i.

See more here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.