2
$\begingroup$

In various papers, I had often seen the logistic regression model for classification problems written in two forms.

$$p(y =\pm1|\mathbf{x},\mathbf{w}) = \sigma(y\mathbf{w}^{T}\mathbf{x}) = \frac{1}{1+\exp(-y\mathbf{w}^T\mathbf{x})} \tag 1$$

and

$$p(y =1|\mathbf{x},\mathbf{w}) = \sigma(\mathbf{w}^{T}\mathbf{x}) = \frac{1}{1+\exp(-\mathbf{w}^T\mathbf{x})} \tag 2$$

The first form has the label being multiplied to the dot product of $\mathbf{w}$ and $\mathbf{x}$ and the second doesn't. Are both the forms are equivalent? If not, how do they differ?

References:

  1. Minka, T.P. (2003). Algorithms for maximum likelihood logistic regression (pdf)
  2. Abu-El-Haija, S. Derivation of logistic regression (pdf)
$\endgroup$
3
  • $\begingroup$ I don't think the first form is correct. After all, this would mean that to determine the probability of whether y is is +/- 1, one would have to know y. $\endgroup$
    – meh
    Dec 22, 2017 at 14:41
  • $\begingroup$ @aginensky i have added some examples where they are used. $\endgroup$
    – Morpheus
    Dec 22, 2017 at 14:44
  • $\begingroup$ @morpheus - You tricked me :) . Your second formula only considers the case $y = 1$. I hadn't noticed that. Of course this means that $P( y = -1) = 1- P(y = 1)) $ and as Lii observed, this gives the correct answer. $\endgroup$
    – meh
    Dec 22, 2017 at 19:09

1 Answer 1

6
$\begingroup$

They are equivalent, so both are correct.

It can be proved simply by checking the probabilities for $Y= +/- 1$

In the first model:

$$P(Y=1 |x,w) = \frac{1}{1+\exp(-w^Tx)} = \frac{\exp(w^Tx)}{1+\exp(w^Tx)}$$

$$P(Y=-1 |x,w) = \frac{1}{1+\exp(w^Tx)} $$

In the second:

$$P(Y=1 |x,w) = \frac{1}{1+\exp(-w^Tx)} = \frac{\exp(w^Tx)}{1+\exp(w^Tx)}$$ $$P(Y=-1 |x,w) = 1 - P(Y=1 |x,w) = \frac{1}{1+\exp(w^Tx)}$$

Equivalent.

Note:

The first model requires $y$ to be labeled with $+/- 1$.

In the second, $y$ can be any label, as long as it's binary, but usually a 1/0 coding is used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.