0
$\begingroup$

Suppose I have data points $(x_1^1, x_2^1), (x_1^2, x_2^2), (x_1^3, x_2^3), \ldots$ in $\mathbf{R}^2$ that fall in one of two classes, $y^i=0$ or $y^i=1$. I can find a linear separator for these points using logistic regression by finding $b_0, b_1, b_2$ that maximize $$P\left(y\;|\; (x_1, x_2)\right) = \frac{1}{1+e^{-(b_0 + b_1x_1 + b_2x_2)}}$$

based on the data. We can do this by gradient ascent, for example. The line that separates the points is given by the set of points $(x_1', x_2')$ that satisfy $$0.5 = \frac{1}{1+e^{-(b_0 + b_1x_1' + b_2x_2')}},$$ which yields $$b_0 + b_1x_1' + b_2x_2' = 0.$$

Right?

Another way to look at this problem is as a regression problem: We can find parameters $\beta_0, \beta_1, \beta_2$ that minimize $$\sum_d (\beta_0 + \beta_1x_1^d + \beta_2x_2^d-y^d)^2.$$ We can do this using the master equations or gradient ascent... So now the line $y=\beta_0 + \beta_1x_1 + \beta_2x_2$ can be viewed in $3d$ as the line that "best" passes through the data? I think the set of points $(x_1', x_2')$ that satisfy $0.5=\beta_0 + \beta_1x_1' + \beta_2x_2'$ will be the same set of points that were found in the first method above, but I'm having trouble seeing why. Thinking about the data as two big clusters in $3d$ seems to be throwing me off. Can someone shed some light on this?

$\endgroup$
  • $\begingroup$ Consider a very basic case. $ b_0=b_1=b_2=0$. Then no points can satisfy your constraint. $\endgroup$ – probabilityislogic Sep 1 '15 at 8:06
  • $\begingroup$ Which constraint are you referring to? $\endgroup$ – Fequish Sep 1 '15 at 12:31
0
$\begingroup$

If you look at the problem as a linear regression problem (linear probit to be correct), the predicted values will not be bound by [1,0]. Therefore points satisfying

0.5= β0+ β1x′1+ β2x′2

will not be the same set of points obtained from the logistic regression. In logistic regression the predicted values are always between 0 and 1.

$\endgroup$
  • $\begingroup$ I can't classify the point $(x_1, x_2)$ by checking whether or not $\beta_0 + \beta_1x_1 + \beta_2x_2 > 0.5$? This seems to be working in the test example I'm playing around with... $\endgroup$ – Fequish Aug 31 '15 at 22:50
  • $\begingroup$ @ fequsih: you could classify using a linear probit, but are they exactly same as the points that you obtained from the logistic classifier? I doubt it. It might be something to do with the data. May be you have perfect separation with your data. $\endgroup$ – subra Aug 31 '15 at 22:56
  • $\begingroup$ My data is not linearly separable. Can you think of a 2d dataset in which both methods would give very different results? $\endgroup$ – Fequish Aug 31 '15 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.