Why is it that logistic regression is considered a linear classifier, but a neural network with a sigmoid activation function and sigmoid output layer is considered non-linear? I am aware that logistic regression is considered linear because its inputs are a linear combination of the original inputs, but couldn't the same be said of the neural network example?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
A neural network with a single sigmoid neuron is also a linear classifier when the output is thresholded for classification. But, more than one layer produces a non-linear one because the decision rule can't be written in the following form: $$\sum w_i x_i+b>\tau$$ where $\tau$ is threshold, $w_i$ are learnable weights and $x_i$ are features. This is why logistic regression is referred as a linear classifier.
-
1$\begingroup$ Maybe to elaborate a little: When talking about the (non)linearity of models, the combination of weights is considered. The use of non-linear function does not lead necessarily to non linear models. (although they are needed to "create" them). In case of a logistic regression only a single sigmoid function is used. But all weights are added linearly before that transformation. In a neural network at least two non-linear transformations are used. No you have a scenario where the combinations in not linear. $\endgroup$– JanoschJun 3, 2022 at 16:00
-
$\begingroup$ So, a neural network with 2 sigmoid units is considered to be non-linear simply because the input to the second sigmoid unit is non-linear due to the non-linear activation of the inputs? $\endgroup$ Jun 3, 2022 at 20:35
-
$\begingroup$ @terribleprogrammer, this has to do with how many hidden layers there are. In the case of logistic regression, there is zero. If there is at least one, you get nonlinearity. $\endgroup$ Jun 4, 2022 at 8:08