6
$\begingroup$

Why is it that logistic regression is considered a linear classifier, but a neural network with a sigmoid activation function and sigmoid output layer is considered non-linear? I am aware that logistic regression is considered linear because its inputs are a linear combination of the original inputs, but couldn't the same be said of the neural network example?

$\endgroup$

1 Answer 1

12
$\begingroup$

A neural network with a single sigmoid neuron is also a linear classifier when the output is thresholded for classification. But, more than one layer produces a non-linear one because the decision rule can't be written in the following form: $$\sum w_i x_i+b>\tau$$ where $\tau$ is threshold, $w_i$ are learnable weights and $x_i$ are features. This is why logistic regression is referred as a linear classifier.

$\endgroup$
3
  • 1
    $\begingroup$ Maybe to elaborate a little: When talking about the (non)linearity of models, the combination of weights is considered. The use of non-linear function does not lead necessarily to non linear models. (although they are needed to "create" them). In case of a logistic regression only a single sigmoid function is used. But all weights are added linearly before that transformation. In a neural network at least two non-linear transformations are used. No you have a scenario where the combinations in not linear. $\endgroup$
    – Janosch
    Commented Jun 3, 2022 at 16:00
  • $\begingroup$ So, a neural network with 2 sigmoid units is considered to be non-linear simply because the input to the second sigmoid unit is non-linear due to the non-linear activation of the inputs? $\endgroup$ Commented Jun 3, 2022 at 20:35
  • $\begingroup$ @terribleprogrammer, this has to do with how many hidden layers there are. In the case of logistic regression, there is zero. If there is at least one, you get nonlinearity. $\endgroup$ Commented Jun 4, 2022 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.