Difference between multiple hidden neurons and hidden layers

Question

I've seen a douzen of these question and I still feel like I'm not really grasping the concept. I've provided a picture that explains some of my concerns:

The picture to the right is acting like an AND-gate, with 3 lines. I keep wondering, is the three lines related to the fact that I have 3 hidden neurons, where the number of hidden nodes decides the number of discernible areas that I can have, but if this interpretation is correct, when would I want to use more hidden layers, what is the point? As a second question, let's presume that I have 3 hidden layers, and I have an activation function in the form of a sigmoid. My question is, is there any area that is too complex that a sigmoid function wouldn't be able to classify it for us? Because it seems to me that you are calculating the weighted sum, and then passing it through the activation function for each layer. As a last question, how does the sigmoid function really work for us, let's say I have like 1 hidden layer, using a sigmoid activation function with 10 hidden neurons, what is really happening, is the sigmoid function giving us building blocks in the form of sigmoid functions that I "build" with? I realize it was a lot of questions but this is really mindboogling me...

Answer 1

Yes, that specific example have three decisions the form $w_{i1}x_1+w_{i2}x_2+b_i\geq0$, which corresponds to the 3 hidden neurons we have. The output layer works like an AND gate. More layers mean more complex decision boundaries, other than combinations of lines; e.g. you can have a boundary like:

$$af(w_{11}x_1+w_{22}x_2+b_1)+bf(w_{21}x_1+w_{22}x_2+b_2)+c\geq0$$

Thinking in 2D for simplicity, of course, you could also form the decision boundaries you need via some combinations of lines. But, number of lines (hidden neurons) to describe such a region might be huge. Let's say you have a dataset in the form of concentric circles, i.e. your decision is $x^2+y^2\geq 1$ for class $1$, or $\leq$ for class $0$. In order to approximate a circle shaped decision boundary, you'd need lots of lines. But, this might be possible using much fewer neurons if you use more hidden layers. Moreover, Universal Approximation Theorem (UAT), is somehow related to what you're asking. It basically states that a single hidden layer with finite number of neurons (but this can be exponentially large) can approximate a wide variety of continuous functions. So, the basic reason of adding (enough) layers is to reduce the number of parameters needed.

For your second question, as I said, one layer can be enough, three layers would be again enough; but if you use enough (which is unknown) neurons. It's not the matter of just using $N$ layers, the neuron population is somewhat crucial. Thinking again in 2D, any region you'd print out and show me will probably a compact subset of $\mathbb{R}^n$, (condition for applying the UAT). And using sigmoid activation has been proven to work with UAT. No matter how many hidden layers you put, with enough neurons it can classify the dataset you put.

For your third question, I think this post about ReLU function can give you some intuition. The case for sigmoid, although harder to imagine, is pretty much similar.