19
$\begingroup$

I have seen two types of logistic loss formulations. We can easily show they are identical, the only difference is the definition of the label $y$.

Formulation/notation 1, $y \in \{0, +1\}$:

$$ L(y,\beta^Tx)=-y\log(p)-(1-y)\log(1-p) $$

where $p=\frac 1 {1+\exp(-\beta^Tx)}$, where the logistic function map a real number $\beta^T x$ to 0,1 interval.

Formulation/notation 2, $y \in \{-1, +1\}$:

$$ L(y,\beta^Tx)=\log(1+\exp{(-y\cdot \beta^Tx})) $$

Choosing a notation is like choosing a language, there are pros and cons to use one or another. What are the pros and cons for these two notations?


My attempts to answer this question is that it seems statistics community likes the first notation and computer science community likes the second notation.

  • First notation can be explained with the term "probability", as the logistic function transform a real number $\beta^Tx$ to 0,1 interval.
  • And second notation is more concise and it is more easy to compare with hinge loss or 0-1 loss.

Am I right? Any other insights?

$\endgroup$
  • 4
    $\begingroup$ I am sure this must have been asked multiple times already. E.g. stats.stackexchange.com/q/145147/5739 $\endgroup$ – StasK Aug 13 '16 at 16:50
  • 1
    $\begingroup$ Why do you say the second notation is easier to compare with hinge loss? Just because it's defined on $\{-1, 1\}$ instead of $\{0, 1\}$, or something else? $\endgroup$ – shadowtalker Aug 27 '16 at 2:39
  • 1
    $\begingroup$ I kinda like the symmetry of the first form, but the linear part is buried pretty deep, so it can be hard to work with. $\endgroup$ – Matthew Drury Aug 27 '16 at 3:08
  • $\begingroup$ @ssdecontrol please check this figure, cs.cmu.edu/~yandongl/loss.html where x axis is $-y\cdot \beta^Tx$, and y axis is loss value. Such definition is convenient to compare with 01 loss, hinge loss, etc. $\endgroup$ – hxd1011 Aug 27 '16 at 15:20
12
$\begingroup$

The short version

  • Yes
  • Yes

The long version

The nice thing about mathematical modeling is that it's flexible. These are indeed equivalent loss functions, but they derive from very different underlying models of the data.

Formula 1

The first notation derives from a Bernoulli probability model for $y$, which is conventionally defined on $\{0, 1\}$. In this model, the outcome/label/class/prediction is represented by a random variable $Y$ that follows a $\mathrm{Bernoulli}(p)$ distribution. Therefore its likelihood is: $$ P(Y = y\ |\ p) = \mathcal L(p; y) = p^y\ (1-p)^{1-y} = \begin{cases}1-p &y=0 \\ p &y=1 \end{cases} $$

for $p\in[0, 1]$. Using 0 and 1 as the indicator values lets us reduce the piecewise function on the far right to a concise expression.

As you've pointed out, you can then link $Y$ to a matrix of input data $x$ by letting $\operatorname{logit} p = \beta^T x$. From here, straightforward algebraic manipulation reveals that $\log \mathcal L(p;y)$ is the same as the first $L(y, \beta^Tx)$ in your question (hint: $(y - 1) = - (1 - y)$). So minimizing log-loss over $\{0, 1\}$ is equivalent to maximum likelihood estimation of a Bernoulli model.

This formulation is also a special case of the generalized linear model, which is formulated as $Y \sim D(\theta),\ g(Y) = \beta^T x$ for an invertible, differentiable function $g$ and a distribution $D$ in the exponential family.

Formula 2

Actually.. I'm not familiar with Formula 2. However, defining $y$ on $\{-1, 1\}$ is standard in the formulation of a support vector machine. Fitting an SVM corresponds to maximizing $$ \max \left(\{0, 1 - y \beta^T x \}\right) + \lambda \|\beta\|^2. $$

This is the Lagrangian form of a constrained optimization problem. It is also an example of a regularized optimization problem with objective function $$ \ell(y, \beta) + \lambda \|\beta\|^2 $$ For some loss function $\ell$ and a scalar hyperparameter $\lambda$ that controls the amount of regularization (also called "shrinkage") applied to $\beta$. Hinge loss is just one of several drop-in possibilities for $\ell$, which also include the second $L(y, \beta^Tx)$ in your question.

$\endgroup$
  • $\begingroup$ In Formula 1, should it not be: $$p^y(1 - p)^{\pmb{1 - y}}$$ $\endgroup$ – glebm Feb 8 '17 at 6:38
7
$\begingroup$

I think @ssdecontrol had a very good answer. I just want to add some comments for the formula 2 for my own question.

$$ L(y,\hat y)=\log(1+\exp{(-y\cdot \hat y})) $$

The reason people like this formulation is that it is very concise, and it removes the "probability interpretation details".

The tricky notation is the $\hat y$, note, $y$ is a binary variable, but $\hat y$ here is a real number. Comparing to formulation 1, we need two additional steps to make it to discrete label, step 1. sigmod function step 2. apply 0.5 threshold.

But without these details are good in terms of we can easily compare it with other classification loss, such as 01 loss or hinge loss. $$ L_{01}(y,\hat y)=I[y \cdot \hat y >0]\\ L_{\text{hinge}}(y,\hat y)=(1-y \cdot \hat y)_+\\ L_{\text{logistic}}(y,\hat y)=\log(1+\exp(-y \cdot \hat y)) $$

enter image description here

Here we plot three loss functions, x axis is $y \cdot \hat y$ and y axis is the loss value. Note, in all above formulas $\hat y$ is a real number, and this number can come from linear form $\beta^Tx$ or other forms. Such notation hides probability details.

$\endgroup$
  • $\begingroup$ I see what you mean about easy comparison $\endgroup$ – shadowtalker Aug 28 '16 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.