I read about two versions of the loss function for logistic regression, which of them is correct and why?

  1. From Machine Learning, Zhou Z.H (in Chinese), with $\beta = (w, b)\text{ and }\beta^Tx=w^Tx +b$:

    $$l(\beta) = \sum\limits_{i=1}^{m}\Big(-y_i\beta^Tx_i+\ln(1+e^{\beta^Tx_i})\Big) \tag 1$$

  2. From my college course, with $z_i = y_if(x_i)=y_i(w^Tx_i + b)$:

    $$L(z_i)=\log(1+e^{-z_i}) \tag 2$$

I know that the first one is an accumulation of all samples and the second one is for a single sample, but I am more curious about the difference in the form of two loss functions. Somehow I have a feeling that they are equivalent.


6 Answers 6


The relationship is as follows: $l(\beta) = \sum_i L(z_i)$.

Define a logistic function as $f(z) = \frac{e^{z}}{1 + e^{z}} = \frac{1}{1+e^{-z}}$. They possess the property that $f(-z) = 1-f(z)$. Or in other words:

$$ \frac{1}{1+e^{z}} = \frac{e^{-z}}{1+e^{-z}}. $$

If you take the reciprocal of both sides, then take the log you get:

$$ \ln(1+e^{z}) = \ln(1+e^{-z}) + z. $$

Subtract $z$ from both sides and you should see this:

$$ -y_i\beta^Tx_i+ln(1+e^{y_i\beta^Tx_i}) = L(z_i). $$


At the moment I am re-reading this answer and am confused about how I got $-y_i\beta^Tx_i+ln(1+e^{\beta^Tx_i})$ to be equal to $-y_i\beta^Tx_i+ln(1+e^{y_i\beta^Tx_i})$. Perhaps there's a typo in the original question.

Edit 2:

In the case that there wasn't a typo in the original question, @ManelMorales appears to be correct to draw attention to the fact that, when $y \in \{-1,1\}$, the probability mass function can be written as $P(Y_i=y_i) = f(y_i\beta^Tx_i)$, due to the property that $f(-z) = 1 - f(z)$. I am re-writing it differently here, because he introduces a new equivocation on the notation $z_i$. The rest follows by taking the negative log-likelihood for each $y$ coding. See his answer below for more details.

  • $\begingroup$ This answer should not be accepted. In the third equation, we can not just replace $z_{i}$ with $y_{i}\cdot<\beta,x_{i}>$ since $y_{i}$ can be $0$ for this LogLoss form. @ManuelMorales is the correct answer and it explicitly points out that the labels we choose determines the LogLoss form $\endgroup$
    – Kuo
    Commented Jul 19, 2021 at 17:01

OP mistakenly believes the relationship between these two functions is due to the number of samples (i.e. single vs all). However, the actual difference is simply how we select our training labels.

In the case of binary classification we may assign the labels $y=\pm1$ or $y=0,1$.

As it has already been stated, the logistic function $\sigma(z)$ is a good choice since it has the form of a probability, i.e. $\sigma(-z)=1-\sigma(z)$ and $\sigma(z)\in (0,1)$ as $z\rightarrow \pm \infty$. If we pick the labels $y=0,1$ we may assign

\begin{equation} \begin{aligned} \mathbb{P}(y=1|z) & =\sigma(z)=\frac{1}{1+e^{-z}}\\ \mathbb{P}(y=0|z) & =1-\sigma(z)=\frac{1}{1+e^{z}}\\ \end{aligned} \end{equation}

which can be written more compactly as $\mathbb{P}(y|z) =\sigma(z)^y(1-\sigma(z))^{1-y}$.

It is easier to maximize the log-likelihood. Maximizing the log-likelihood is the same as minimizing the negative log-likelihood. For $m$ samples $\{x_i,y_i\}$, after taking the natural logarithm and some simplification, we will find out:

\begin{equation} \begin{aligned} l(z)=-\log\big(\prod_i^m\mathbb{P}(y_i|z_i)\big)=-\sum_i^m\log\big(\mathbb{P}(y_i|z_i)\big)=\sum_i^m-y_iz_i+\log(1+e^{z_i}) \end{aligned} \end{equation}

Full derivation and additional information can be found on this jupyter notebook. On the other hand, we may have instead used the labels $y=\pm 1$. It is pretty obvious then that we can assign

\begin{equation} \mathbb{P}(y|z)=\sigma(yz). \end{equation}

It is also obvious that $\mathbb{P}(y=0|z)=\mathbb{P}(y=-1|z)=\sigma(-z)$. Following the same steps as before we minimize in this case the loss function

\begin{equation} \begin{aligned} L(z)=-\log\big(\prod_j^m\mathbb{P}(y_j|z_j)\big)=-\sum_j^m\log\big(\mathbb{P}(y_j|z_j)\big)=\sum_j^m\log(1+e^{-y_jz_j}) \end{aligned} \end{equation}

Where the last step follows after we take the reciprocal which is induced by the negative sign. While we should not equate these two forms, given that in each form $y$ takes different values, nevertheless these two are equivalent.

While there may be fundamental reasons as to why we have two different forms (see Why there are two different logistic loss formulation / notations?), one reason to choose the former is for practical considerations. In the former we can use the property $\partial \sigma(z) / \partial z=\sigma(z)(1-\sigma(z))$ to trivially calculate $\nabla l(z)$ and $\nabla^2l(z)$, both of which are needed for convergence analysis (i.e. to determine the convexity of the loss function by calculating the Hessian).

  • $\begingroup$ Is logistic loss function convex? $\endgroup$
    – user85361
    Commented Jun 18, 2017 at 6:12
  • 2
    $\begingroup$ Log reg $l(z)$ IS convex, but not $\alpha$-convex. Thus we can't place a bound on how long gradient descent takes to converge. We can adjust the form of $l$ to make it strongly convex by adding a regularization term: with positive constant $\lambda$ define our new function to be $l'(z)=l(z)+\lambda\|z\|^2$ s.t $l'(z)$ is $\lambda$-strongly convex and we can now prove the convergence bound of $l'$. Unfortunately, we are now minimizing a different function! Luckily, we can show that the value of the optimum of the regularized function is close to the value of the optimum of the original. $\endgroup$ Commented Jul 8, 2017 at 19:57
  • $\begingroup$ The notebook you referred has gone, I got another proof: statlect.com/fundamentals-of-statistics/… $\endgroup$
    – Domi.Zhang
    Commented Jan 15, 2018 at 7:18
  • 4
    $\begingroup$ I found this to be the most helpful answer. $\endgroup$
    – mohit6up
    Commented Jan 19, 2018 at 16:00
  • $\begingroup$ @ManuelMorales Do you have a link to the regularized function's optimum value being close to the original? $\endgroup$ Commented Jun 8, 2018 at 22:06

I learned the loss function for logistic regression as follows.

Logistic regression performs binary classification, and so the label outputs are binary, 0 or 1. Let $P(y=1|x)$ be the probability that the binary output $y$ is 1 given the input feature vector $x$. The coefficients $w$ are the weights that the algorithm is trying to learn.

$$P(y=1|x) = \frac{1}{1 + e^{-w^{T}x}}$$

Because logistic regression is binary, the probability $P(y=0|x)$ is simply 1 minus the term above.

$$P(y=0|x) = 1- \frac{1}{1 + e^{-w^{T}x}}$$

The loss function $J(w)$ is the sum of (A) the output $y=1$ multiplied by $P(y=1)$ and (B) the output $y=0$ multiplied by $P(y=0)$ for one training example, summed over $m$ training examples.

$$J(w) = \sum_{i=1}^{m} y^{(i)} \log P(y=1) + (1 - y^{(i)}) \log P(y=0)$$

where $y^{(i)}$ indicates the $i^{th}$ label in your training data. If a training instance has a label of $1$, then $y^{(i)}=1$, leaving the left summand in place but making the right summand with $1-y^{(i)}$ become $0$. On the other hand, if a training instance has $y=0$, then the right summand with the term $1-y^{(i)}$ remains in place, but the left summand becomes $0$. Log probability is used for ease of calculation.

If we then replace $P(y=1)$ and $P(y=0)$ with the earlier expressions, then we get:

$$J(w) = \sum_{i=1}^{m} y^{(i)} \log \left(\frac{1}{1 + e^{-w^{T}x}}\right) + (1 - y^{(i)}) \log \left(1- \frac{1}{1 + e^{-w^{T}x}}\right)$$

You can read more about this form in these Stanford lecture notes.

  • $\begingroup$ This answer also provides some relevant perspective here. $\endgroup$
    – GeoMatt22
    Commented Dec 11, 2016 at 23:53
  • 7
    $\begingroup$ The expression you have is not a loss (to be minimized), but rather a log-likelihood (to be maximized). $\endgroup$
    – xenocyon
    Commented Jun 7, 2017 at 22:08
  • 2
    $\begingroup$ @xenocyon true - this same formulation is typically written with a negative sign applied to the full summation. $\endgroup$ Commented Nov 17, 2017 at 23:05

Instead of Mean Squared Error, we use a cost function called Cross-Entropy, also known as Log Loss. Cross-entropy loss can be divided into two separate cost functions: one for y=1 and one for y=0.

\begin{align}\newcommand{\Cost}{{\rm Cost}}\newcommand{\if}{{\rm if}} j(\theta) &= \frac 1 m \sum_{i=1}^m \Cost(h_\theta(x^{(i)}), y^{(i)}) & & \\ \Cost(h_\theta(x), y) &= -\log(h_\theta(x)) & \if\ y &= 1 \\ \Cost(h_\theta(x), y) &= -\log(1-h_\theta(x)) & \if\ y &= 0 \end{align}

When we put them together we have:

$$ j(\theta) = \frac 1 m \sum_{i=1}^m \big[y^{(i)}\log(h_\theta(x^{(i)})) + (1-y^{(i)})\log(1-h_\theta(x)^{(i)}) \big] $$

Multiplying by $y$ and $(1−y)$ in the above equation is a sneaky trick that let’s us use the same equation to solve for both $y=1$ and $y=0$ cases. If $y=0$, the first side cancels out. If $y=1$, the second side cancels out. In both cases we only perform the operation we need to perform.

If you don't want to use a for loop, you can try a vectorized form of the equation above

\begin{align} h &= g(X\theta) \\ J(\theta) &= \frac 1 m \cdot \big(-y^T\log(h)-(1-y)^T\log(1-h)\big) \end{align}

The entire explanation can be view on Machine Learning Cheatsheet.


They are the same functions.

In the first one, $y_i$ is either $0$ or $1$. While in the second, $y_i$ is either $-1$ or $1$.

The second one can be derived from the first one, as the probabilities in the second function can be written as only one equation, that is, the sigmoid function of $z_i y_i$ ($z_i$ is the linear combinations of features of observation $i$; $y_i$ is $-1$ or $1$ based on the fact of observation $i$).


Both loss functions can be derived from the generalized linear model paradigm. I'll start with the case: $P(Y=1 |X = x) = \phi$ and $P(Y=-1 |X = x) = 1 - \phi.$ Now, combine them as follows $$P(Y=y |X=x) = \phi^{\frac{1+y}{2}} (1 - \phi)^{\frac{1-y}{2}} = e^{\ln(\frac{\phi}{1-\phi})\frac{y}{2} + \frac{1}{2}\ln(\phi(1-\phi))}.$$ Accordingly, the probability mass function is in the form of the probability mass function of a random variable in the exponential family. In turn, the natural parameter $\eta = \ln(\frac{\phi}{1-\phi}),$ which implies $\phi = \frac{e^{\eta}}{1 + e^{\eta}} = \frac{1}{1 + e^{-\eta}}$ and $1 - \phi = \frac{1}{1 + e^{\eta}}.$

Next, consider $$\ln(P(Y=y |X=x)) = \frac{1+y}{2}\ln(\phi) + \frac{1-y}{2}\ln(1-\phi) = ... = -\frac{1}{2}\big((1+y)\ln(1 + e^{-\eta}) + (1-y)\ln(1 + e^{\eta}) \big).$$

This gives two cases. If $y=-1,$ then the natural logarithm of the probability mass function becomes $$\ln(P(Y=-1 |X=x)) = -\frac{1}{2}\big( 2 \ln(1+e^{\eta})\big) = -\ln(1+e^{\eta}) = -\ln(1+e^{-y\eta}).$$ Otherwise, if $y=1,$ then the natural logarithm of the probability mass function becomes $$\ln(P(Y=1 |X=x)) = -\frac{1}{2}\big( 2 \ln(1+e^{-\eta})\big) = -\ln(1+e^{-\eta}) = -\ln(1+e^{-y\eta}).$$

Then, if there are $m$ training examples, the log-likelihood becomes $$\ell(\theta) = \sum_{i=1}^{m} \ln(P(Y=y^{(i)} |X=x^{(i)})) = -\sum_{i=1}^{m} \ln(1+e^{-y^{(i)}\theta^{t}x^{(i)}}),$$ where the natural parameter is assumed to be the inner product $\theta^{t}x^{(i)}$ in the case of a generalized linear model. Lastly, the loss function becomes $$J(\theta) = - \ell(\theta) = \sum_{i=1}^{m} \ln(1+e^{-y^{(i)}\theta^{t}x^{(i)}}).$$

The other loss function can be derived in this way by imposing that the probability mass function be a Bernoulli probability mass function. This derivation is ubiquitous so it is not repeated here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.