# Gradient for logistic loss function

I would ask a question related to this one.

I found an example of writing custom loss function for xgboost here:

loglossobj <- function(preds, dtrain) {
# dtrain is the internal format of the training data
# We extract the labels from the training data
labels <- getinfo(dtrain, "label")
# We compute the 1st and 2nd gradient, as grad and hess
preds <- 1/(1 + exp(-preds))
hess <- preds * (1 - preds)
# Return the result as a list
}


Logistic loss function is

$$log(1+e^{-yP})$$

where $P$ is log-odds and $y$ is labels (0 or 1).

My question is: how we can get gradient (first derivative) simply equal to difference between true values and predicted probabilities (calculated from log-odds as preds <- 1/(1 + exp(-preds)))?

• You should use squared error loss to achieve that. Your notation is confusing and should be defined in the post. If $p$ is the predicted risk, then $(y-p)^2$ loss is what you want. I'm confused because we never use $p$ to mean the log-odds. – AdamO Jun 16 '16 at 18:19
• $p$ was fixed to capital $P$. It is log-odds, and it is clearly marked in the question. I know that gradient for loss function $(y-f(x))^2$ is $f(x)-y$, but it is squred loss, not logistic. – Ogurtsov Jun 16 '16 at 18:41
• When you say the "gradient", what gradient do you mean? The gradient of the loss? It's a simple mathematical relationship that if the derivative of an expression is a linear difference, then the expression is a quadratic difference, or squared error loss. – AdamO Jun 16 '16 at 19:36
• Yes, it is all about gradient of the loss. It is simple, when loss function is squared error. In this case loss function is logistic loss (en.wikipedia.org/wiki/LogitBoost), and I can't find correspondence between gradient of this function and given code example. – Ogurtsov Jun 17 '16 at 3:10

My answer for my question: yes, it can be shown that gradient for logistic loss is equal to difference between true values and predicted probabilities. Brief explanation was found here.

First, logistic loss is just negative log-likelihood, so we can start with expression for log-likelihood (p. 74 - this expression is log-likelihood itself, not negative log-likelihood):

$$L=y_{i}\cdot log(p_{i})+(1-y_{i})\cdot log(1-p_{i})$$

$p_{i}$ is logistic function: $p_{i}=\frac{1}{1+e^{-\hat{y}_{i}}}$, where $\hat{y}_{i}$ is predicted values before logistic transformation (i.e., log-odds):

$$L=y_{i}\cdot log\left(\frac{1}{1+e^{-\hat{y}_{i}}}\right)+(1-y_{i})\cdot log\left(\frac{e^{-\hat{y}_{i}}}{1+e^{-\hat{y}_{i}}}\right)$$

First derivative obtained using Wolfram Alpha:

$${L}'=\frac{y_{i}-(1-y_{i})\cdot e^{\hat{y}_{i}}}{1+e^{\hat{y}_{i}}}$$

After multiplying by $\frac{e^{-\hat{y}_{i}}}{e^{-\hat{y}_{i}}}$:

$${L}'=\frac{y_{i}\cdot e^{-\hat{y}_{i}}+y_{i}-1}{1+e^{-\hat{y}_{i}}}= \frac{y_{i}\cdot (1+e^{-\hat{y}_{i}})}{1+e^{-\hat{y}_{i}}}-\frac{1}{1+e^{-\hat{y}_{i}}}=y_{i}-p_{i}$$

After changing sign we have expression for gradient of logistic loss function:

$$p_{i}-y_{i}$$

• What you're calling $\hat{y}$ here is not a prediction of $y$, but a linear combination of predictors. In generalized linear modeling we use the notation $\nu$ and call this term the "linear predictor". Your derivative of the loglikelihood (score) is wrong, there should be a squared term in the denominator, since bernoullis form an exponential likelihood. The score should be of the form $\frac{1}{p_i(1-p_i)}(y_i - p_i)$ – AdamO Jul 5 '16 at 18:04

AdamO is correct, if you just want the gradient of the logistic loss (what the op asked for in the title), then it needs a 1/p(1-p). Unfortunately people from the DL community for some reason assume logistic loss to always be bundled with a sigmoid, and pack their gradients together and call that the logistic loss gradient (the internet is filled with posts asserting this). Since the gradient of sigmoid happens to be p(1-p) it eliminates the 1/p(1-p) of the logistic loss gradient. But if you are implementing SGD (walking back the layers), and applying the sigmoid gradient when you get to the sigmoid, then you need to start with the actual logistic loss gradient -- which has a 1/p(1-p).