Why there are two different logistic loss formulation / notations?

Question

I have seen two types of logistic loss formulations. We can easily show they are identical, the only difference is the definition of the label $y$.

Formulation/notation 1, $y \in \{0, +1\}$:

$$ L(y,\beta^Tx)=-y\log(p)-(1-y)\log(1-p) $$

where $p=\frac 1 {1+\exp(-\beta^Tx)}$, where the logistic function map a real number $\beta^T x$ to 0,1 interval.

Formulation/notation 2, $y \in \{-1, +1\}$:

$$ L(y,\beta^Tx)=\log(1+\exp{(-y\cdot \beta^Tx})) $$

Choosing a notation is like choosing a language, there are pros and cons to use one or another. What are the pros and cons for these two notations?

---

My attempts to answer this question is that it seems statistics community likes the first notation and computer science community likes the second notation.

• First notation can be explained with the term "probability", as the logistic function transform a real number $\beta^Tx$ to 0,1 interval.
• And second notation is more concise and it is more easy to compare with hinge loss or 0-1 loss.

Am I right? Any other insights?

Answer 1

The short version

• Yes
• Yes

The long version

The nice thing about mathematical modeling is that it's flexible. These are indeed equivalent loss functions, but they derive from very different underlying models of the data.

Formula 1

The first notation derives from a Bernoulli probability model for $y$, which is conventionally defined on $\{0, 1\}$. In this model, the outcome/label/class/prediction is represented by a random variable $Y$ that follows a $\mathrm{Bernoulli}(p)$ distribution. Therefore its likelihood is: $$ P(Y = y\ |\ p) = \mathcal L(p; y) = p^y\ (1-p)^{1-y} = \begin{cases}1-p &y=0 \\ p &y=1 \end{cases} $$

for $p\in[0, 1]$. Using 0 and 1 as the indicator values lets us reduce the piecewise function on the far right to a concise expression.

As you've pointed out, you can then link $Y$ to a matrix of input data $x$ by letting $\operatorname{logit} p = \beta^T x$. From here, straightforward algebraic manipulation reveals that $\log \mathcal L(p;y)$ is the same as the first $L(y, \beta^Tx)$ in your question (hint: $(y - 1) = - (1 - y)$). So minimizing log-loss over $\{0, 1\}$ is equivalent to maximum likelihood estimation of a Bernoulli model.

This formulation is also a special case of the generalized linear model, which is formulated as $Y \sim D(\theta),\ g(Y) = \beta^T x$ for an invertible, differentiable function $g$ and a distribution $D$ in the exponential family.

Formula 2

Actually.. I'm not familiar with Formula 2. However, defining $y$ on $\{-1, 1\}$ is standard in the formulation of a support vector machine. Fitting an SVM corresponds to maximizing $$ \max \left(\{0, 1 - y \beta^T x \}\right) + \lambda \|\beta\|^2. $$

This is the Lagrangian form of a constrained optimization problem. It is also an example of a regularized optimization problem with objective function $$ \ell(y, \beta) + \lambda \|\beta\|^2 $$ For some loss function $\ell$ and a scalar hyperparameter $\lambda$ that controls the amount of regularization (also called "shrinkage") applied to $\beta$. Hinge loss is just one of several drop-in possibilities for $\ell$, which also include the second $L(y, \beta^Tx)$ in your question.

Answer 2

I think @ssdecontrol had a very good answer. I just want to add some comments for the formula 2 for my own question.

$$ L(y,\hat y)=\log(1+\exp{(-y\cdot \hat y})) $$

The reason people like this formulation is that it is very concise, and it removes the "probability interpretation details".

The tricky notation is the $\hat y$, note, $y$ is a binary variable, but $\hat y$ here is a real number. Comparing to formulation 1, we need two additional steps to make it to discrete label, step 1. sigmod function step 2. apply 0.5 threshold.

But without these details are good in terms of we can easily compare it with other classification loss, such as 01 loss or hinge loss. $$ L_{01}(y,\hat y)=I[y \cdot \hat y >0]\\ L_{\text{hinge}}(y,\hat y)=(1-y \cdot \hat y)_+\\ L_{\text{logistic}}(y,\hat y)=\log(1+\exp(-y \cdot \hat y)) $$

Here we plot three loss functions, x axis is $y \cdot \hat y$ and y axis is the loss value. Note, in all above formulas $\hat y$ is a real number, and this number can come from linear form $\beta^Tx$ or other forms. Such notation hides probability details.