6
$\begingroup$

There are two ways to write the objective function (negative log-likelihood) for logistic regression:

  1. Let

\begin{align} & p=P(y^{(1)}=1\mid x)= \frac{1}{1+e^{-\beta^\mathrm{T}x}}\\ & P(y^{(1)}=0\mid x)= 1-p=\frac{1}{1+e^{\beta^\mathrm{T}x}} \end{align}

The negative log-likelihood will be \begin{align} l=&-\sum_i\left[y^{(1)}_i\log p+(1-y^{(1)}_i)\log(1-p)\right]\\ =&\sum_i\left[y^{(1)}_i\beta^{\mathrm{T}}x_i-\log(1+e^{\beta^{\mathrm{T}}x_i})\right] \end{align}

  1. Let

\begin{align} & p=P(y^{(2)}=1\mid x)= \frac{1}{1+e^{-\beta^\mathrm{T}x}}\\ & P(y^{(2)}=-1\mid x)= 1-p=\frac{1}{1+e^{\beta^\mathrm{T}x}} \end{align}

which can be written into a single formula

$$ P(y^{(2)}\mid x)=\frac{1}{1+e^{-y^{(2)}\beta^\mathrm{T}x}} $$

so the negative log-likelihood will be

$$ l=\sum_i\log(1+e^{-y^{(2)}_i\beta^\mathrm{T}x_i}) $$

I think these two forms are equivalent because how we denote the labels should not matter. Can I algebraically transform one to the other? I tried $y^{(1)}_i=(1+y^{(2)}_i)/2$ in the first formulation but it seems it doesn't do the trick.

$\endgroup$
3
  • 2
    $\begingroup$ Those two forms are equivalent in that you label two categories the way you want. You could use "carrot" and "turnip" as well. Given that those are labels, rather than numbers, mathematical transforms do not apply. (Note that the second log-likelihood can be written differently, e.g., with $y_i^{(2)}$ outside the log. $\endgroup$ – Xi'an Apr 7 '15 at 13:09
  • $\begingroup$ @Xi'an but we use them as numbers. When $y=0\mbox{ or }1$, we use Bernoulli distribution; when $y=\pm 1$, we use the sign. $\endgroup$ – ziyuang Apr 7 '15 at 13:16
  • $\begingroup$ The problem is the probability model for $y$ is different: the mean of a balanced (p=0.5) outcome is 0.5 in the first scenario and 0 in the second. If you calculate the fisher scoring for a estimating equation using a modified logit link: $g = \log(y^{(2)}/2)/(1-\log(y^{(2)}/2))$ you could show they are the same. Otherwise they are not. $\endgroup$ – AdamO Feb 20 '18 at 15:14
2
$\begingroup$

I think you lost (or added) a minus sign in one of the formulations (maybe going from log-lik to neg-log-lik?).

Using $z=\beta x$, if we write the losses as:

  • $l_{01} = -y_{01} z + \log (1+e^z)$
  • $l_{\pm1} = \log (1+e^{-y_{\pm1}z})$

Then when $y_{01} = y_{\pm1} = 1$, we have: $$-z + \log (1 + e^z) \quad \text{and} \quad \log (1 + e^{-z})$$

which we can show are equal with a bit of algebra.

When $y_{01} = 0$ and $y_{\pm1} = -1$, we have: $$\log (1 + e^z) \quad \text{and} \quad \log (1 + e^z)$$ which is what we need.

While this doesn't give you the substitution you were looking for, it does show the equivalence. See also a related answer: https://stats.stackexchange.com/a/279698/1704.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.