# Logistic regressions questions about line fitting vs. probabilistic interperetation

Suppose I have data points $(x_1^1, x_2^1), (x_1^2, x_2^2), (x_1^3, x_2^3), \ldots$ in $\mathbf{R}^2$ that fall in one of two classes, $y^i=0$ or $y^i=1$. I can find a linear separator for these points using logistic regression by finding $b_0, b_1, b_2$ that maximize $$P\left(y\;|\; (x_1, x_2)\right) = \frac{1}{1+e^{-(b_0 + b_1x_1 + b_2x_2)}}$$

based on the data. We can do this by gradient ascent, for example. The line that separates the points is given by the set of points $(x_1', x_2')$ that satisfy $$0.5 = \frac{1}{1+e^{-(b_0 + b_1x_1' + b_2x_2')}},$$ which yields $$b_0 + b_1x_1' + b_2x_2' = 0.$$

Right?

Another way to look at this problem is as a regression problem: We can find parameters $\beta_0, \beta_1, \beta_2$ that minimize $$\sum_d (\beta_0 + \beta_1x_1^d + \beta_2x_2^d-y^d)^2.$$ We can do this using the master equations or gradient ascent... So now the line $y=\beta_0 + \beta_1x_1 + \beta_2x_2$ can be viewed in $3d$ as the line that "best" passes through the data? I think the set of points $(x_1', x_2')$ that satisfy $0.5=\beta_0 + \beta_1x_1' + \beta_2x_2'$ will be the same set of points that were found in the first method above, but I'm having trouble seeing why. Thinking about the data as two big clusters in $3d$ seems to be throwing me off. Can someone shed some light on this?

• Consider a very basic case. $b_0=b_1=b_2=0$. Then no points can satisfy your constraint. Commented Sep 1, 2015 at 8:06
• Which constraint are you referring to? Commented Sep 1, 2015 at 12:31

If you look at the problem as a linear regression problem (linear probit to be correct), the predicted values will not be bound by [1,0]. Therefore points satisfying

0.5= β0+ β1x′1+ β2x′2

will not be the same set of points obtained from the logistic regression. In logistic regression the predicted values are always between 0 and 1.

• I can't classify the point $(x_1, x_2)$ by checking whether or not $\beta_0 + \beta_1x_1 + \beta_2x_2 > 0.5$? This seems to be working in the test example I'm playing around with... Commented Aug 31, 2015 at 22:50
• @ fequsih: you could classify using a linear probit, but are they exactly same as the points that you obtained from the logistic classifier? I doubt it. It might be something to do with the data. May be you have perfect separation with your data. Commented Aug 31, 2015 at 22:56
• My data is not linearly separable. Can you think of a 2d dataset in which both methods would give very different results? Commented Aug 31, 2015 at 23:17