I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script.
I've simplified the function to take numpy arrays, and generated y_hat
and y_true
which are a sample of the values used in the script.
Here is the simplified example:
import numpy as np
def loglikelihoodloss(y_hat, y_true):
prob = 1.0 / (1.0 + np.exp(-y_hat))
grad = prob - y_true
hess = prob * (1.0 - prob)
return grad, hess
y_hat = np.array([1.80087972, -1.82414818, -1.82414818, 1.80087972, -2.08465433,
-1.82414818, -1.82414818, 1.80087972, -1.82414818, -1.82414818])
y_true = np.array([1., 0., 0., 1., 0., 0., 0., 1., 0., 0.])
loglikelihoodloss(y_hat, y_true)
The log loss function is the sum of $y\ln\left(p\right)+\left(1-y\right)\ln\left(1-p\right)$ where $p = \dfrac{1}{(1 + e^{-x})}$.
The gradient (with respect to p) is then $\dfrac{p-y}{\left(p-1\right)p}$ however in the code its $p -y$.
Likewise the second derivative (with respect to p) is $\dfrac{\left(y-p\right)p+y\left(p-1\right)}{\left(p-1\right)^2p^2}$ however in the code it is $p(1-p)$.
How are the equations equal?