12
$\begingroup$

I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script.

I've simplified the function to take numpy arrays, and generated y_hat and y_true which are a sample of the values used in the script.

Here is the simplified example:

import numpy as np


def loglikelihoodloss(y_hat, y_true):
    prob = 1.0 / (1.0 + np.exp(-y_hat))
    grad = prob - y_true
    hess = prob * (1.0 - prob)
    return grad, hess

y_hat = np.array([1.80087972, -1.82414818, -1.82414818,  1.80087972, -2.08465433,
                  -1.82414818, -1.82414818,  1.80087972, -1.82414818, -1.82414818])
y_true = np.array([1.,  0.,  0.,  1.,  0.,  0.,  0.,  1.,  0.,  0.])

loglikelihoodloss(y_hat, y_true)

The log loss function is the sum of $y\ln\left(p\right)+\left(1-y\right)\ln\left(1-p\right)$ where $p = \dfrac{1}{(1 + e^{-x})}$.

The gradient (with respect to p) is then $\dfrac{p-y}{\left(p-1\right)p}$ however in the code its $p -y$.

Likewise the second derivative (with respect to p) is $\dfrac{\left(y-p\right)p+y\left(p-1\right)}{\left(p-1\right)^2p^2}$ however in the code it is $p(1-p)$.

How are the equations equal?

$\endgroup$
1
  • $\begingroup$ what is x in e^(-x)? $\endgroup$
    – Maths12
    Sep 2, 2020 at 13:49

1 Answer 1

19
$\begingroup$

The derivatives are with respect to $x$ (or y_hat in the code) instead of $p$.

As you've already derived (Edit: as Simon.H mentioned, since the actual loss should be the negative log likelihood, so I've changed the sign of your result) $$\frac{\partial f}{\partial p}=\frac{p-y}{\left(1-p\right)p},$$ and the derivative of sigmoid is $$\frac{\partial p}{\partial x}=p(1-p),$$ so $$\frac{\partial f}{\partial x}=\frac{\partial f}{\partial p}\frac{\partial p}{\partial x}=p-y,$$ and the second order derivative $$\frac{\partial^2 f}{\partial x^2}=\frac{\partial}{\partial x}(\frac{\partial f}{\partial x})=\frac{\partial}{\partial x}(p-y)=\frac{\partial p}{\partial x}=p(1-p).$$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.