In logistic regression The following function produces a linear decision boundary

1) h(x)=sigmoid(w1.x1 + w2.x2 +...+bias) i.e. h(x)=sigmoid(z(x))

Eventhough there is a non linear activation like sigmoid, since the input features are all linear, the decision boundary z(x)=0 is also linear.

2) whereas if

h(x)=sigmoid(w1.x1^2 + w2.x2^2 + w3.x1.x2 + w4.x1 + w5.x2 +...+bias) i.e h(x)=sigmoid(z(x))

now the decision boundary z(x)=0 is nonlinear since the input terms are nonlinear.


In Neural networks, the multiple hidden neurons just output a final linear combination like the first scenario since the input terms were linear (x1,x2) not (x1,x2,x1^2,x2^2,x1,x2). So how can it create a non linear decision boundary?


I think the key point is that: when you combine multiple linear boundaries, you get a nonlinear boundary

For example, if I want to separate the grey circles from the black circles (below), I could draw a single boundary (which is what logistic regression would be akin to):
1 boundary

But a single boundary doesnt do the best job. Alternatively, I could combine multiple decision boundaries, to do an even better job:
2 boundaries

In a neural network, you can sort of think of each hidden node as a linear-like decision boundary; the network can combine them to form very nonlinear boundaries (for example, a network with 2 hidden nodes might produce the following):
combined boundaries

And you can combine as many hidden nodes as you like; here's an example of a network with 5 sigmoid hidden nodes solving an XOR-like problem: enter image description here

Shikhar Sharma has a very useful blog post that does a good job of explaining to.

  • $\begingroup$ wow that was a surprisingly detailed answer $\endgroup$ – Lelouche Lamperouge Apr 30 '20 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.