Two equivalent forms of logistic regression

Question

There are two ways to write the objective function (negative log-likelihood) for logistic regression:

1. Let

\begin{align} & p=P(y^{(1)}=1\mid x)= \frac{1}{1+e^{-\beta^\mathrm{T}x}}\\ & P(y^{(1)}=0\mid x)= 1-p=\frac{1}{1+e^{\beta^\mathrm{T}x}} \end{align}

The negative log-likelihood will be \begin{align} l=&-\sum_i\left[y^{(1)}_i\log p+(1-y^{(1)}_i)\log(1-p)\right]\\ =&\sum_i\left[y^{(1)}_i\beta^{\mathrm{T}}x_i-\log(1+e^{\beta^{\mathrm{T}}x_i})\right] \end{align}

2. Let

\begin{align} & p=P(y^{(2)}=1\mid x)= \frac{1}{1+e^{-\beta^\mathrm{T}x}}\\ & P(y^{(2)}=-1\mid x)= 1-p=\frac{1}{1+e^{\beta^\mathrm{T}x}} \end{align}

which can be written into a single formula

$$ P(y^{(2)}\mid x)=\frac{1}{1+e^{-y^{(2)}\beta^\mathrm{T}x}} $$

so the negative log-likelihood will be

$$ l=\sum_i\log(1+e^{-y^{(2)}_i\beta^\mathrm{T}x_i}) $$

I think these two forms are equivalent because how we denote the labels should not matter. Can I algebraically transform one to the other? I tried $y^{(1)}_i=(1+y^{(2)}_i)/2$ in the first formulation but it seems it doesn't do the trick.

Answer 1

I think you lost (or added) a minus sign in one of the formulations (maybe going from log-lik to neg-log-lik?).

Using $z=\beta x$, if we write the losses as:

• $l_{01} = -y_{01} z + \log (1+e^z)$
• $l_{\pm1} = \log (1+e^{-y_{\pm1}z})$

Then when $y_{01} = y_{\pm1} = 1$, we have: $$-z + \log (1 + e^z) \quad \text{and} \quad \log (1 + e^{-z})$$

which we can show are equal with a bit of algebra.

When $y_{01} = 0$ and $y_{\pm1} = -1$, we have: $$\log (1 + e^z) \quad \text{and} \quad \log (1 + e^z)$$ which is what we need.

While this doesn't give you the substitution you were looking for, it does show the equivalence. See also a related answer: https://stats.stackexchange.com/a/279698/1704.