3
$\begingroup$

I'm currently enrolled in a regression models course through Coursera and am having trouble conceptually understanding logistic regression. I've attached a picture of "where" I'm having trouble.

Linear vs logistic regression

Linear

$$ RW_i = b_0 + b_1RS_i + e_i $$ or $$ E[RW_i|RS_i, b_0,b_1] = b_0+b_1RS $$

Logistic

$$ {\rm Pr}(RW_i|RS_i,b_0,b_1) = \frac{\exp(b_0+b_1RS)}{1+\exp(b_0+b_1RS)} $$ or $$ \log\bigg(\frac{{\rm Pr}(RW_i|RS_i,b_0,b_1)}{1-{\rm Pr}(RW_i|RS_i,b_0,b_1)}\bigg) = b_0+b_1RS $$

In this particular example, the lecture is about modeling Bernoulli outcomes. I understand that when modeling binary data in a linear fashion, the expected value of the outcome is the probability of said outcome (first two formulas in the picture). What I'm having trouble with is where the second two formulas came from and how they can be interchanged (in the lecture, he says if we "invert/work with" [the 3rd formula] it can be rewritten as the 4th formula) I'd like to know the connection between the last two equations and how they should be interpreted.

$\endgroup$
  • 1
    $\begingroup$ Maybe a duplicate: stats.stackexchange.com/questions/71176/… $\endgroup$ – kjetil b halvorsen Aug 31 '17 at 14:29
  • 2
    $\begingroup$ Possible duplicate of Intuition behind logistic regression $\endgroup$ – Alecos Papadopoulos Aug 31 '17 at 14:56
  • 1
    $\begingroup$ @kjetilbhalvorsen Since the OP asked for intuition, certainly the answer in the link you provided, presenting the "threshold response model" answers "where all come from". I voted to close as duplicate. $\endgroup$ – Alecos Papadopoulos Aug 31 '17 at 14:57
  • $\begingroup$ How to move between the 3rd & 4th formulas is not covered in the linked thread, but it is something that can be elucidated & would be on topic. So I'm voting to leave open. $\endgroup$ – gung - Reinstate Monica Aug 31 '17 at 16:58
  • $\begingroup$ thank you @gung, thats exactly what I wanted to see worked out. $\endgroup$ – Marshall McQuillen Aug 31 '17 at 18:31
7
$\begingroup$

(@Gijs is right that this is just algebra, but perhaps it will help you to see it worked out.)

Consider that "$\exp(b_0+b_1RS)$" is an odds of some event, and that "${\rm Pr}(RW_i|RS_i,b_0,b_1)$" is the probability of the same event (cf., Interpretation of simple predictions to odds ratios in logistic regression). To make it easier to see what's going on, we can replace these complicated expressions with "$o$" and "$p$", respectively.

Now, recognize that:
$$ p=\frac{o}{1+o} \qquad\qquad o=\frac{p}{1-p} $$ If you substitute the full expressions into the formula on the left, you will get your third equation.

On the other hand, if we substitute the full expressions into the right formula, you would get this:
$$ \exp(b_0+b_1RS) = \bigg(\frac{{\rm Pr}(RW_i|RS_i,b_0,b_1)}{1-{\rm Pr}(RW_i|RS_i,b_0,b_1)}\bigg) $$ which isn't quite the same. To get to your fourth formula, take the log of both sides, and then just switch the right hand side and the left hand side:
\begin{align} \log(\exp(b_0+b_1RS)) &= \log\bigg(\frac{{\rm Pr}(RW_i|RS_i,b_0,b_1)}{1-{\rm Pr}(RW_i|RS_i,b_0,b_1)}\bigg) \\[8pt] \log\bigg(\frac{{\rm Pr}(RW_i|RS_i,b_0,b_1)}{1-{\rm Pr}(RW_i|RS_i,b_0,b_1)}\bigg) &= b_0+b_1RS \end{align}

Thus, all we need to establish is the fact that the "$o$"s in the right equation above are the same as the "$o$" in the left equation, and likewise for the "$p$"s between the two equations. What is potentially unintuitive is that you will need to take the reciprocal of both sides at a couple of points. It's a bit tedious to write out the algebra, but it doesn't take too many steps:
\begin{align} p &= \frac{o}{1+o} &\text{right formula}& \\[8pt] \frac 1 p &= \frac{1+o}{o} &\text{taking reciprocals}& \\[8pt] \frac 1 p &= \frac 1 o + \frac o o &\text{separate out }\frac o o& \\[8pt] \frac 1 p &= \frac 1 o + 1 &\frac o o = 1& \\[8pt] \frac 1 p - 1 &= \frac 1 o &\text{subtracting 1}& \\[8pt] \frac 1 p - \frac p p &= \frac 1 o &1 = \frac p p& \\[8pt] \frac{1-p}{p} &= \frac 1 o &\text{simplifying}& \\[8pt] \frac{p}{1-p} &= o &\text{taking reciprocals}& \\[8pt] o &= \frac{p}{1-p} &\text{left formula}& \end{align}

You might also find What is the difference between logistic and logit regression? helpful.

$\endgroup$
4
$\begingroup$

It's just algebra to show that if

$$ \frac{e^x}{1 + e^x} = a $$

then

$$ \log(\frac{a}{1 - a}) = x $$

and this explains the equivalence between the last two formulas.

The first of these formulas translates the outcome of the regression line ($x$ in that formula), which could be any real number, to a number between zero and one ($a$ in that formula). This is called a "link" function. This number between zero and one is used as the modelled probability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.