# Why is logistic regression a linear classifier?

Since we are using the logistic function to transform a linear combination of the input into a non-linear output, how can logistic regression be considered a linear classifier?

Linear regression is just like a neural network without the hidden layer, so why are neural networks considered non-linear classifiers and logistic regression is linear?

• Transforming "a linear combination of the input into a non-linear output" is a basic part of the definition of a Linear Classifier. That reduces this question to the second part, which amounts to demonstrating that Neural Networks cannot generally be expressed as linear classifiers.
– whuber
Commented Apr 12, 2014 at 19:51
• @whuber: How do you explain the fact that a logistic regression model can take polynomial predictor variables (e.g. $w_1 \cdot x_1^2 + w_2 \cdot x_2^3$) to produce a non-linear decision boundary? Is that still a linear classifier? Commented Jun 25, 2016 at 22:33
• @Stack The concept of "linear classifier" appears to originate with the concept of a linear model. "Linearity" in a model can take on several forms, as described at stats.stackexchange.com/a/148713. If we accept the Wikipedia characterization of linear classifiers, then your polynomial example would be viewed as nonlinear in terms of the given "features" $x_1$ and $x_2$ but it would be linear in terms of the features $x_1^2$ and $x_2^3$. This distinction provides a useful way to exploit the properties of linearity.
– whuber
Commented Jul 12, 2016 at 13:12
• I'm still a bit confused about the question is the decision boundary of a logistic classifier linear? I've followed the Andrew Ng machine learning course on Coursera and he mentioned the following: ![enter image description here](i.sstatic.net/gHxfr.png) So actually it seems to me there is no one answer it depends on the linearity or non-linearity of the decision boundary, that depends on the Hypothesis function defined as Htheta(X) where X is the input and Theta is the variables of our problem. Does it make sense for you? Commented Nov 30, 2016 at 9:48

Logistic regression is linear in the sense that the predictions can be written as $$\hat{p} = \frac{1}{1 + e^{-\hat{\mu}}}, \text{ where } \hat{\mu} = \hat{\theta} \cdot x.$$ Thus, the prediction can be written in terms of $\hat{\mu}$, which is a linear function of $x$. (More precisely, the predicted log-odds is a linear function of $x$.)

Conversely, there is no way to summarize the output of a neural network in terms of a linear function of $x$, and that is why neural networks are called non-linear.

Also, for logistic regression, the decision boundary $\{x:\hat{p} = 0.5\}$ is linear: it's the solution to $\hat{\theta} \cdot x = 0$. The decision boundary of a neural network is in general not linear.

• You answer is the most clear and uncomplicated to me so far. But I'm a bit confused. Some people say that the predicated log-odds is a linear function of $x$ and others say it's a linear function of $\theta$. So?! Commented Apr 16, 2014 at 16:56
• The predicted log-odds $\hat{\theta} \cdot x$ is linear in both $\hat{\theta}$ and $x$. But usually we are most interested in the fact that the log-odds is linear in $x$, because this implies that the decision boundary is linear in $x$ space. Commented Apr 16, 2014 at 17:58
• I've been using the definition that a classifier is linear if its decision boundary is linear in $x$ space. This is not the same as the predicted probabilities being linear in $x$ (which would be impossible apart from trivial cases, since probabilities must lie between 0 and 1). Commented Apr 16, 2014 at 22:01
• @Pegah I know this is old, but: Logistic regression has a linear decision boundary. The ouptut itself is not linear of course, its logistic. Depending on which side of the line a point falls, the total output will approach (but never reach) 0 or 1 respectively. And to add to Stefan Wagners answer: The last sentence is not totally correct, a neural network is non-linear when it contains non-linear activations or ouput functions. But it can be linear as well (in case no non-linearities were added). Commented Aug 14, 2017 at 23:12
• I still don't understand the difference between how logistic regression is linear, but NNs aren't. Can't you write the output of a neural network as $\hat p = w_1\sigma(x_1)+w_2\sigma(x_2)+w_3$, which is similar to how the OP of this answer wrote the output of logistic regression? Equivalently, $\hat p = w_1(\frac{1}{1+exp(-\theta^Tx_1)}) + w_2(\frac{1}{1+exp(-\theta^Tx_2)})+w_3\\$. So my confusion is that if the answer's equation is considered to be a linear function of x, what makes this not a linear function of x? Commented Aug 21, 2017 at 14:57

As Stefan Wagner notes, the decision boundary for a logistic classifier is linear. (The classifier needs the inputs to be linearly separable.) I wanted to expand on the math for this in case it's not obvious.

The decision boundary is the set of x such that $${1 \over {1 + e^{-{\theta \cdot x}}}} = 0.5$$

A little bit of algebra shows that this is equivalent to $${1 = e^{-{\theta \cdot x}}}$$

and, taking the natural log of both sides,

$$0 = -\theta \cdot x = -\sum\limits_{i=0}^{n} \theta_i x_i$$

so the decision boundary is linear.

The reason the decision boundary for a neural network is not linear is because there are two layers of sigmoid functions in the neural network: one in each of the output nodes plus an additional sigmoid function to combine and threshold the results of each output node.

• Actually, you can get a non-linear decision boundary with only one layer having an activation. See the standard example of an XOR with a 2-layer feed-forward network. Commented Mar 19, 2018 at 18:48
• Logistic regression is neither linear nor is it a classifier. The idea of a "decision boundary" has little to do with logistic regression, which is instead a direct probability estimation method that separates predictions from decision. Commented Nov 18, 2020 at 13:48
• A neural network can have a single layer and then it can be equivalent to a logistic regression model (depends on the choice of activation function). As a matter of fact, neural networks historically started by being single layered (e.g. Perceptron). Commented Oct 10, 2022 at 18:26

It we have two classes, $C_{0}$ and $C_{1}$, then we can express the conditional probability as, $$P(C_{0}|x) = \frac{P(x|C_{0})P(C_{0})}{P(x)}$$ applying the Bayes' theorem, $$P(C_{0}|x) = \frac{P(x|C_{0})P(C_{0})}{P(x|C_{0})P(C_{0})+P(x|C_{1})P(C_{1})} = \frac{1}{1+ \exp\left(-\log\frac{P(x|C_{0})}{P(x|C_{1})}-\log \frac{P(C_{0})}{P(C_{1})}\right)}$$ the denominator is expressed as $1+e^{\omega x}$.

Under which conditions reduces the first expression to a linear term?. If you consider the exponential family (a canonical form for the exponential distributions like Gauß or Poisson), $$P(x|C_{i}) = \exp \left(\frac{\theta_{i} x -b(\theta_{i})}{a(\phi)}+c(x,\phi)\right)$$ then you end up having a linear form, $$\log\frac{P(x|C_{0})}{P(x|C_{1})} = \left[ (\theta_{0}-\theta_{1})x - b(\theta_{0})+b(\theta_{1}) \right]/a(\phi)$$

Notice that we assume that both distributions belong to the same family and have the same dispersion parameters. But, under that assumption, the logistic regression can model the probabilities for the whole family of exponential distributions.