2
$\begingroup$

I'm learning logistic regression on Coursera.

In the class, we say the logistic regression model is $h_\theta(x) = p(y=1) =\frac{1}{1+e^{-\theta^Tx}}$. When making predictions, we say that $y=1$ if $h_\theta(x) \ge .5$ and $y=0$ otherwise. The cost function we use to find the parameters $\theta$ given the training set is $J(\theta)=\frac{1}{2m}\sum (-y^{(i)}h_\theta(x^{(i)}) - (1-y^{(i)})(1-h_\theta(x^{(i)})))$.

But couldn't we predict the exact same thing using linear regression? Our linear model is $a = \theta^TX$ and we output $y=1$ if $a \ge0$ and $y=0$ otherwise. We could make our cost function $J(\theta)=\frac{1}{2m}\sum (-y^{(i)}g(a) - (1-y^{(i)})(1-g(a)))$ where $g$ is the sigmoid function. Using linear regression with this new cost function and a threshold of $0$ would give us exactly the same predictions as logistic regression.

So why do we use logistic regression instead of linear regression with a new cost function? Is it just because the probability in logistic regression is more easily interpretable than the real number outcome in linear regression? Or is there another reason?

$\endgroup$

marked as duplicate by Xi'an, Tim, Michael Chernick, John, kjetil b halvorsen Feb 14 '17 at 9:28

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ This is no longer linear regression when you use a non-linear function $g$ in the least-squares minimisation. The logistic function $g$ is one possible choice, the probit function another one, and the sigmoid function yet another one. Any cdf could be used to this purpose. $\endgroup$ – Xi'an Feb 14 '17 at 5:28
3
$\begingroup$

You hint at the correct reason in your last paragraph, it is because logistic regression predicts conditional probabilities.

I would venture the strong optinion that, regardless of what you learned in class, this

When making predictions, we say that $y=1$ if $h_\theta(x) \ge .5$ and $y=0$ otherwise.

is incorrect, especially, but not uniquely, in the context of logistic regression.

Logistic regression is a probabilistic model, once trained you can interpret predictions from a logistic regression as the conditional probabilites

$$ h_\theta(x) = P(y = 1 \mid x) $$

In practice, having an estimate of these conditional probabilities is much, much more useful than hard classifying new data points. With the probabilities you gain the power to compute expectations of many statistics of interest in your problem (say profit, revenue, loss), or simulate new scenarios by drawing from distributions based on these estimated probabilities.

Since you can, if needed, hard classify data by thresholding the probabilities, you lose nothing by estimating them. This also allows you the freedom to choose a threshold to obtain optimal results in whatever classification task is at hand, strict adherence to a threshold of $0.5$ is generally a sign of an inexperienced data scientist.

So, the use of logistic regression to hard classify a new datapoint as either $y = 0$ or $y = 1$ by thresholding the estimated probabilities is an extra layer added on in addition to the regression itself. It is not correct to view hard classification as the job or intent of the regression. A hard classification may be your eventual intent, and part of your execution of the plan for achieving this goal may indeed be to fit a logistic regression, but the job of the regression is all and only to estimate conditional probabilities.

Finally, we don't use linear regression because it simply does not fulfill the same role. The least squares criterion for fitting a linear regression does not respect the role of the predictions as conditional probabilities, while logistic regression maximizes the likelihood of the training data with respect to the predicted conditional probabilities. Additionally, the predictions from linear regression can be any real number, which negates their use as probabilities.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.