11
$\begingroup$

I would ask a question related to this one.

I found an example of writing custom loss function for xgboost here:

loglossobj <- function(preds, dtrain) {
  # dtrain is the internal format of the training data
  # We extract the labels from the training data
  labels <- getinfo(dtrain, "label")
  # We compute the 1st and 2nd gradient, as grad and hess
  preds <- 1/(1 + exp(-preds))
  grad <- preds - labels
  hess <- preds * (1 - preds)
  # Return the result as a list
  return(list(grad = grad, hess = hess))
}

Logistic loss function is

$$log(1+e^{-yP})$$

where $P$ is log-odds and $y$ is labels (0 or 1).

My question is: how we can get gradient (first derivative) simply equal to difference between true values and predicted probabilities (calculated from log-odds as preds <- 1/(1 + exp(-preds)))?

$\endgroup$
  • $\begingroup$ You should use squared error loss to achieve that. Your notation is confusing and should be defined in the post. If $p$ is the predicted risk, then $(y-p)^2$ loss is what you want. I'm confused because we never use $p$ to mean the log-odds. $\endgroup$ – AdamO Jun 16 '16 at 18:19
  • $\begingroup$ $p$ was fixed to capital $P$. It is log-odds, and it is clearly marked in the question. I know that gradient for loss function $(y-f(x))^2$ is $f(x)-y$, but it is squred loss, not logistic. $\endgroup$ – Ogurtsov Jun 16 '16 at 18:41
  • $\begingroup$ When you say the "gradient", what gradient do you mean? The gradient of the loss? It's a simple mathematical relationship that if the derivative of an expression is a linear difference, then the expression is a quadratic difference, or squared error loss. $\endgroup$ – AdamO Jun 16 '16 at 19:36
  • $\begingroup$ Yes, it is all about gradient of the loss. It is simple, when loss function is squared error. In this case loss function is logistic loss (en.wikipedia.org/wiki/LogitBoost), and I can't find correspondence between gradient of this function and given code example. $\endgroup$ – Ogurtsov Jun 17 '16 at 3:10
17
$\begingroup$

My answer for my question: yes, it can be shown that gradient for logistic loss is equal to difference between true values and predicted probabilities. Brief explanation was found here.

First, logistic loss is just negative log-likelihood, so we can start with expression for log-likelihood (p. 74 - this expression is log-likelihood itself, not negative log-likelihood):

$$L=y_{i}\cdot log(p_{i})+(1-y_{i})\cdot log(1-p_{i})$$

$p_{i}$ is logistic function: $p_{i}=\frac{1}{1+e^{-\hat{y}_{i}}}$, where $\hat{y}_{i}$ is predicted values before logistic transformation (i.e., log-odds):

$$L=y_{i}\cdot log\left(\frac{1}{1+e^{-\hat{y}_{i}}}\right)+(1-y_{i})\cdot log\left(\frac{e^{-\hat{y}_{i}}}{1+e^{-\hat{y}_{i}}}\right)$$

First derivative obtained using Wolfram Alpha:

$${L}'=\frac{y_{i}-(1-y_{i})\cdot e^{\hat{y}_{i}}}{1+e^{\hat{y}_{i}}}$$

After multiplying by $\frac{e^{-\hat{y}_{i}}}{e^{-\hat{y}_{i}}}$:

$${L}'=\frac{y_{i}\cdot e^{-\hat{y}_{i}}+y_{i}-1}{1+e^{-\hat{y}_{i}}}= \frac{y_{i}\cdot (1+e^{-\hat{y}_{i}})}{1+e^{-\hat{y}_{i}}}-\frac{1}{1+e^{-\hat{y}_{i}}}=y_{i}-p_{i}$$

After changing sign we have expression for gradient of logistic loss function:

$$p_{i}-y_{i}$$

$\endgroup$
  • 2
    $\begingroup$ What you're calling $\hat{y}$ here is not a prediction of $y$, but a linear combination of predictors. In generalized linear modeling we use the notation $\nu$ and call this term the "linear predictor". Your derivative of the loglikelihood (score) is wrong, there should be a squared term in the denominator, since bernoullis form an exponential likelihood. The score should be of the form $\frac{1}{p_i(1-p_i)}(y_i - p_i)$ $\endgroup$ – AdamO Jul 5 '16 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.