46
$\begingroup$

Since we are using the logistic function to transform a linear combination of the input into a non-linear output, how can logistic regression be considered a linear classifier?

Linear regression is just like a neural network without the hidden layer, so why are neural networks considered non-linear classifiers and logistic regression is linear?

$\endgroup$
  • 6
    $\begingroup$ Transforming "a linear combination of the input into a non-linear output" is a basic part of the definition of a Linear Classifier. That reduces this question to the second part, which amounts to demonstrating that Neural Networks cannot generally be expressed as linear classifiers. $\endgroup$ – whuber Apr 12 '14 at 19:51
  • $\begingroup$ @whuber: How do you explain the fact that a logistic regression model can take polynomial predictor variables (e.g. $w_1 \cdot x_1^2 + w_2 \cdot x_2^3$) to produce a non-linear decision boundary? Is that still a linear classifier? $\endgroup$ – stackoverflowuser2010 Jun 25 '16 at 22:33
  • 2
    $\begingroup$ @Stack The concept of "linear classifier" appears to originate with the concept of a linear model. "Linearity" in a model can take on several forms, as described at stats.stackexchange.com/a/148713. If we accept the Wikipedia characterization of linear classifiers, then your polynomial example would be viewed as nonlinear in terms of the given "features" $x_1$ and $x_2$ but it would be linear in terms of the features $x_1^2$ and $x_2^3$. This distinction provides a useful way to exploit the properties of linearity. $\endgroup$ – whuber Jul 12 '16 at 13:12
  • $\begingroup$ I'm still a bit confused about the question is the decision boundary of a logistic classifier linear? I've followed the Andrew Ng machine learning course on Coursera and he mentioned the following: ![enter image description here](i.stack.imgur.com/gHxfr.png) So actually it seems to me there is no one answer it depends on the linearity or non-linearity of the decision boundary, that depends on the Hypothesis function defined as Htheta(X) where X is the input and Theta is the variables of our problem. Does it make sense for you? $\endgroup$ – brokensword Nov 30 '16 at 9:48
36
$\begingroup$

Logistic regression is linear in the sense that the predictions can be written as $$ \hat{p} = \frac{1}{1 + e^{-\hat{\mu}}}, \text{ where } \hat{\mu} = \hat{\theta} \cdot x. $$ Thus, the prediction can be written in terms of $\hat{\mu}$, which is a linear function of $x$. (More precisely, the predicted log-odds is a linear function of $x$.)

Conversely, there is no way to summarize the output of a neural network in terms of a linear function of $x$, and that is why neural networks are called non-linear.

Also, for logistic regression, the decision boundary $\{x:\hat{p} = 0.5\}$ is linear: it's the solution to $\hat{\theta} \cdot x = 0$. The decision boundary of a neural network is in general not linear.

$\endgroup$
  • 1
    $\begingroup$ You answer is the most clear and uncomplicated to me so far. But I'm a bit confused. Some people say that the predicated log-odds is a linear function of $x$ and others say it's a linear function of $\theta$. So?! $\endgroup$ – Jack Twain Apr 16 '14 at 16:56
  • $\begingroup$ then also by your explanation. Can we say that the predication of the neural network is a linear function of the last hidden layer's activations? $\endgroup$ – Jack Twain Apr 16 '14 at 17:01
  • 1
    $\begingroup$ The predicted log-odds $\hat{\theta} \cdot x$ is linear in both $\hat{\theta}$ and $x$. But usually we are most interested in the fact that the log-odds is linear in $x$, because this implies that the decision boundary is linear in $x$ space. $\endgroup$ – Stefan Wager Apr 16 '14 at 17:58
  • 2
    $\begingroup$ I've been using the definition that a classifier is linear if its decision boundary is linear in $x$ space. This is not the same as the predicted probabilities being linear in $x$ (which would be impossible apart from trivial cases, since probabilities must lie between 0 and 1). $\endgroup$ – Stefan Wager Apr 16 '14 at 22:01
  • 2
    $\begingroup$ @Pegah I know this is old, but: Logistic regression has a linear decision boundary. The ouptut itself is not linear of course, its logistic. Depending on which side of the line a point falls, the total output will approach (but never reach) 0 or 1 respectively. And to add to Stefan Wagners answer: The last sentence is not totally correct, a neural network is non-linear when it contains non-linear activations or ouput functions. But it can be linear as well (in case no non-linearities were added). $\endgroup$ – Chris Aug 14 '17 at 23:12
18
$\begingroup$

As Stefan Wagner notes, the decision boundary for a logistic classifier is linear. (The classifier needs the inputs to be linearly separable.) I wanted to expand on the math for this in case it's not obvious.

The decision boundary is the set of x such that $${1 \over {1 + e^{-{\theta \cdot x}}}} = 0.5$$

A little bit of algebra shows that this is equivalent to $${1 = e^{-{\theta \cdot x}}}$$

and, taking the natural log of both sides,

$$0 = -\theta \cdot x = -\sum\limits_{i=0}^{n} \theta_i x_i$$

so the decision boundary is linear.

The reason the decision boundary for a neural network is not linear is because there are two layers of sigmoid functions in the neural network: one in each of the output nodes plus an additional sigmoid function to combine and threshold the results of each output node.

$\endgroup$
  • 1
    $\begingroup$ Actually, you can get a non-linear decision boundary with only one layer having an activation. See the standard example of an XOR with a 2-layer feed-forward network. $\endgroup$ – James Hirschorn Mar 19 '18 at 18:48
3
$\begingroup$

It we have two classes, $C_{0}$ and $C_{1}$, then we can express the conditional probability as, $$ P(C_{0}|x) = \frac{P(x|C_{0})P(C_{0})}{P(x)} $$ applying the Bayes' theorem, $$ P(C_{0}|x) = \frac{P(x|C_{0})P(C_{0})}{P(x|C_{0})P(C_{0})+P(x|C_{1})P(C_{1})} = \frac{1}{1+ \exp\left(-\log\frac{P(x|C_{0})}{P(x|C_{1})}-\log \frac{P(C_{0})}{P(C_{1})}\right)} $$ the denominator is expressed as $1+e^{\omega x}$.

Under which conditions reduces the first expression to a linear term?. If you consider the exponential family (a canonical form for the exponential distributions like Gauß or Poisson), $$ P(x|C_{i}) = \exp \left(\frac{\theta_{i} x -b(\theta_{i})}{a(\phi)}+c(x,\phi)\right) $$ then you end up having a linear form, $$ \log\frac{P(x|C_{0})}{P(x|C_{1})} = \left[ (\theta_{0}-\theta_{1})x - b(\theta_{0})+b(\theta_{1}) \right]/a(\phi) $$

Notice that we assume that both distributions belong to the same family and have the same dispersion parameters. But, under that assumption, the logistic regression can model the probabilities for the whole family of exponential distributions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.