I am having trouble to understand the loss function scikit-learn uses to fit logistic regression, which can be found here.

Specifically I have problem with the second term. It seems very different from the usual MLE criterion. Can someone give me some hint where this comes from?

$$\mathop {\min{\mkern 1mu} }\limits_{w,c} \frac{1}{2}{w^T}w + C\sum\limits_{i = 1}^n {\log } (\exp ( - {y_i}(X_i^Tw + c)) + 1)$$

I think usually the log likelihood of a logistic regression is something like below. Clearly the first term of below is missing from the scikit-learn objective function.

$$LLH=\sum_{i=1}^n \left[{y_i}(X_i^Tw + c) - \ln\{1+\exp(X_i^Tw + c)\} \right]$$

  • $\begingroup$ I've tried all of the conversion methods that are listed on this page, but none of them worked for me. This answer on a different post is well explained. stats.stackexchange.com/a/279698 $\endgroup$
    – Futa Arai
    Oct 3, 2021 at 1:50

3 Answers 3


These two are actually (almost) equivalent because of the following property of the logistic function:

$$ \sigma(x) = \frac{1}{1+\exp(-x)} = \frac{\exp(x)}{\exp(x)+1} $$


$$ \sum_{i=1}^n \log ( 1 + \exp( -y_i (X_i^T w + c) ) ) \\ = \sum_{i=1}^n \log \left[ (\exp( y_i (X_i^T w + c) ) + 1) \exp( -y_i (X_i^T w + c) ) \right] \\ = -\sum_{i=1}^n \left[ y_i (X_i^T w + c) - \log (\exp( y_i (X_i^T w + c) ) + 1) \right] $$

Note, though, that your formula doesn't have $y_i$ in the "log part", while this one does. (I guess this is a typo)

UPD 2023: Notice that I've assumed signed targets $y \in \{-1, +1\}$. If you instead assume the target is binary $y \in \{0, 1\}$ then you can obtain the second formula exactly, see other answers for details.


I don't think that the lack of $y_i$ is a typo:

The usual log-loss (cross-entropy loss) is: $$-\sum_i [y_i \log(p_i) + (1-y_i) \log(1 - p_i)],$$ where $p_i = \sigma(X^T_i \omega + c)$, and $\sigma(x) = 1/(1+e^{-x})$ is the logistic function.

From there, $$-\sum_i [y_i \log(p_i) + (1-y_i) \log(1 - p_i)] \\ = -\sum_i [y_i \log\left(\frac{p_i}{1-p_i}\right) + \log(1 - p_i)] \\ = -\sum_i [y_i \left( X^T_i \omega + c \right) + \log(1 - p_i)] \\ = -\sum_i [y_i \left( X^T_i \omega + c \right) - \log\left(1 + \exp({X^T_i \omega + c})\right)].$$ This matches the LLH expression given in the original post, without the $y_i$ factor in the exponential.

  • $\begingroup$ Yes, however notice the first formula in the question assumes signed targets $y \in \{-1, +1\}$ while yours assumes binary targets $y \in \{0, 1\}$ $\endgroup$ Jul 17 at 6:55

It is just a matter of the definition of $y_i$. Defining $y_i$ and $\tilde y_i$ such that $y_i \in \{0, 1\}$ and $\tilde y_i \in \{-1, 1\}$ ($\tilde y_i = 2y_i -1$), and using $p_i = \sigma({X^T_i \omega + c})$ and $1- \sigma(x) = \sigma(-x)$, you get

$$-\sum_i [y_i \log(p_i) + (1-y_i) \log(1 - p_i)] = \sum_i \log\left(1 + \exp(-\tilde y_i({X^T_i \omega + c}))\right).$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.