# Gradient for logistic loss function

I would ask a question related to this one.

I found an example of writing custom loss function for xgboost here:

loglossobj <- function(preds, dtrain) {
# dtrain is the internal format of the training data
# We extract the labels from the training data
labels <- getinfo(dtrain, "label")
# We compute the 1st and 2nd gradient, as grad and hess
preds <- 1/(1 + exp(-preds))
hess <- preds * (1 - preds)
# Return the result as a list
}


Logistic loss function is

$$log(1+e^{-yP})$$

where $P$ is log-odds and $y$ is labels (0 or 1).

My question is: how we can get gradient (first derivative) simply equal to difference between true values and predicted probabilities (calculated from log-odds as preds <- 1/(1 + exp(-preds)))?

• You should use squared error loss to achieve that. Your notation is confusing and should be defined in the post. If $p$ is the predicted risk, then $(y-p)^2$ loss is what you want. I'm confused because we never use $p$ to mean the log-odds. – AdamO Jun 16 '16 at 18:19
• $p$ was fixed to capital $P$. It is log-odds, and it is clearly marked in the question. I know that gradient for loss function $(y-f(x))^2$ is $f(x)-y$, but it is squred loss, not logistic. – Ogurtsov Jun 16 '16 at 18:41
• When you say the "gradient", what gradient do you mean? The gradient of the loss? It's a simple mathematical relationship that if the derivative of an expression is a linear difference, then the expression is a quadratic difference, or squared error loss. – AdamO Jun 16 '16 at 19:36
• Yes, it is all about gradient of the loss. It is simple, when loss function is squared error. In this case loss function is logistic loss (en.wikipedia.org/wiki/LogitBoost), and I can't find correspondence between gradient of this function and given code example. – Ogurtsov Jun 17 '16 at 3:10

My answer for my question: yes, it can be shown that gradient for logistic loss is equal to difference between true values and predicted probabilities. Brief explanation was found here.

First, logistic loss is just negative log-likelihood, so we can start with expression for log-likelihood (p. 74 - this expression is log-likelihood itself, not negative log-likelihood):

$$L=y_{i}\cdot log(p_{i})+(1-y_{i})\cdot log(1-p_{i})$$

$p_{i}$ is logistic function: $p_{i}=\frac{1}{1+e^{-\hat{y}_{i}}}$, where $\hat{y}_{i}$ is predicted values before logistic transformation (i.e., log-odds):

$$L=y_{i}\cdot log\left(\frac{1}{1+e^{-\hat{y}_{i}}}\right)+(1-y_{i})\cdot log\left(\frac{e^{-\hat{y}_{i}}}{1+e^{-\hat{y}_{i}}}\right)$$

First derivative obtained using Wolfram Alpha:

$${L}'=\frac{y_{i}-(1-y_{i})\cdot e^{\hat{y}_{i}}}{1+e^{\hat{y}_{i}}}$$

After multiplying by $\frac{e^{-\hat{y}_{i}}}{e^{-\hat{y}_{i}}}$:

$${L}'=\frac{y_{i}\cdot e^{-\hat{y}_{i}}+y_{i}-1}{1+e^{-\hat{y}_{i}}}= \frac{y_{i}\cdot (1+e^{-\hat{y}_{i}})}{1+e^{-\hat{y}_{i}}}-\frac{1}{1+e^{-\hat{y}_{i}}}=y_{i}-p_{i}$$

After changing sign we have expression for gradient of logistic loss function:

$$p_{i}-y_{i}$$

• What you're calling $\hat{y}$ here is not a prediction of $y$, but a linear combination of predictors. In generalized linear modeling we use the notation $\nu$ and call this term the "linear predictor". Your derivative of the loglikelihood (score) is wrong, there should be a squared term in the denominator, since bernoullis form an exponential likelihood. The score should be of the form $\frac{1}{p_i(1-p_i)}(y_i - p_i)$ – AdamO Jul 5 '16 at 18:04