How can a neural network produce non linear decision boundary?

Question

In logistic regression The following function produces a linear decision boundary

1) h(x)=sigmoid(w1.x1 + w2.x2 +...+bias) i.e. h(x)=sigmoid(z(x))

Eventhough there is a non linear activation like sigmoid, since the input features are all linear, the decision boundary z(x)=0 is also linear.

2) whereas if

h(x)=sigmoid(w1.x1^2 + w2.x2^2 + w3.x1.x2 + w4.x1 + w5.x2 +...+bias) i.e h(x)=sigmoid(z(x))

now the decision boundary z(x)=0 is nonlinear since the input terms are nonlinear.

Now

In Neural networks, the multiple hidden neurons just output a final linear combination like the first scenario since the input terms were linear (x1,x2) not (x1,x2,x1^2,x2^2,x1,x2). So how can it create a non linear decision boundary?

Answer 1

I think the key point is that: when you combine multiple linear boundaries, you get a nonlinear boundary

For example, if I want to separate the grey circles from the black circles (below), I could draw a single boundary (which is what logistic regression would be akin to):

But a single boundary doesnt do the best job. Alternatively, I could combine multiple decision boundaries, to do an even better job:

In a neural network, you can sort of think of each hidden node as a linear-like decision boundary; the network can combine them to form very nonlinear boundaries (for example, a network with 2 hidden nodes might produce the following):

And you can combine as many hidden nodes as you like; here's an example of a network with 5 sigmoid hidden nodes solving an XOR-like problem:

Shikhar Sharma has a very useful blog post that does a good job of explaining to.