In logistic regression The following function produces a linear decision boundary

1) h(x)=sigmoid(w1.x1 + w2.x2 +...+bias) i.e. h(x)=sigmoid(z(x))

Eventhough there is a non linear activation like sigmoid, since the input features are all linear, the decision boundary z(x)=0 is also linear.

2) whereas if

h(x)=sigmoid(w1.x1^2 + w2.x2^2 + w3.x1.x2 + w4.x1 + w5.x2 +...+bias) i.e h(x)=sigmoid(z(x))

now the decision boundary z(x)=0 is nonlinear since the input terms are nonlinear.


In Neural networks, the multiple hidden neurons just output a final linear combination like the first scenario since the input terms were linear (x1,x2) not (x1,x2,x1^2,x2^2,x1,x2). So how can it create a non linear decision boundary?


1 Answer 1


I think the key point is that: when you combine multiple linear boundaries, you get a nonlinear boundary

For example, if I want to separate the grey circles from the black circles (below), I could draw a single boundary (which is what logistic regression would be akin to):
1 boundary

But a single boundary doesnt do the best job. Alternatively, I could combine multiple decision boundaries, to do an even better job:
2 boundaries

In a neural network, you can sort of think of each hidden node as a linear-like decision boundary; the network can combine them to form very nonlinear boundaries (for example, a network with 2 hidden nodes might produce the following):
combined boundaries

And you can combine as many hidden nodes as you like; here's an example of a network with 5 sigmoid hidden nodes solving an XOR-like problem: enter image description here

Shikhar Sharma has a very useful blog post that does a good job of explaining to.

  • $\begingroup$ wow that was a surprisingly detailed answer $\endgroup$ Commented Apr 30, 2020 at 15:04
  • $\begingroup$ @Matt I didn't get how could we get non-linear boundaries using multiple linear boundaries. $\endgroup$
    – F.C. Akhi
    Commented Oct 25, 2022 at 16:02
  • $\begingroup$ @Matt How could I attach my code and ask questions here? $\endgroup$
    – F.C. Akhi
    Commented Oct 25, 2022 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.