Gradient for logistic loss function

Question

I would ask a question related to this one.

I found an example of writing custom loss function for xgboost here:

loglossobj <- function(preds, dtrain) {
  # dtrain is the internal format of the training data
  # We extract the labels from the training data
  labels <- getinfo(dtrain, "label")
  # We compute the 1st and 2nd gradient, as grad and hess
  preds <- 1/(1 + exp(-preds))
  grad <- preds - labels
  hess <- preds * (1 - preds)
  # Return the result as a list
  return(list(grad = grad, hess = hess))
}

Logistic loss function is

$$log(1+e^{-yP})$$

where $P$ is log-odds and $y$ is labels (0 or 1).

My question is: how we can get gradient (first derivative) simply equal to difference between true values and predicted probabilities (calculated from log-odds as preds <- 1/(1 + exp(-preds)))?

Answer 1

My answer for my question: yes, it can be shown that gradient for logistic loss is equal to difference between true values and predicted probabilities. Brief explanation was found here.

First, logistic loss is just negative log-likelihood, so we can start with expression for log-likelihood (p. 74 - this expression is log-likelihood itself, not negative log-likelihood):

$$L=y_{i}\cdot log(p_{i})+(1-y_{i})\cdot log(1-p_{i})$$

$p_{i}$ is logistic function: $p_{i}=\frac{1}{1+e^{-\hat{y}_{i}}}$, where $\hat{y}_{i}$ is predicted values before logistic transformation (i.e., log-odds):

$$L=y_{i}\cdot log\left(\frac{1}{1+e^{-\hat{y}_{i}}}\right)+(1-y_{i})\cdot log\left(\frac{e^{-\hat{y}_{i}}}{1+e^{-\hat{y}_{i}}}\right)$$

First derivative obtained using Wolfram Alpha:

$${L}'=\frac{y_{i}-(1-y_{i})\cdot e^{\hat{y}_{i}}}{1+e^{\hat{y}_{i}}}$$

After multiplying by $\frac{e^{-\hat{y}_{i}}}{e^{-\hat{y}_{i}}}$:

$${L}'=\frac{y_{i}\cdot e^{-\hat{y}_{i}}+y_{i}-1}{1+e^{-\hat{y}_{i}}}= \frac{y_{i}\cdot (1+e^{-\hat{y}_{i}})}{1+e^{-\hat{y}_{i}}}-\frac{1}{1+e^{-\hat{y}_{i}}}=y_{i}-p_{i}$$

After changing sign we have expression for gradient of logistic loss function:

$$p_{i}-y_{i}$$

Answer 2

AdamO is correct, if you just want the gradient of the logistic loss (what the op asked for in the title), then it needs a 1/p(1-p). Unfortunately people from the DL community for some reason assume logistic loss to always be bundled with a sigmoid, and pack their gradients together and call that the logistic loss gradient (the internet is filled with posts asserting this). Since the gradient of sigmoid happens to be p(1-p) it eliminates the 1/p(1-p) of the logistic loss gradient. But if you are implementing SGD (walking back the layers), and applying the sigmoid gradient when you get to the sigmoid, then you need to start with the actual logistic loss gradient -- which has a 1/p(1-p).