What is the purpose of having multiple neurons in each layer?

Question

I started a deep learning course (introductory one). But I have two years experience working with ML projects on my own time. This course is just for me to understand deep learning at a more fundamental level.

Starting with a single layer perceptron, it makes sense. We have a linear function to act as a hypothesis. Then that output is fed to an activation function that adds non-linearity and normalises outputs.

That totally makes sense to me. Now my understanding so far:

1. When we have multiple neurons in a layer, the connections to the neuron represents some weight. The neuron itself is just an activation function. (That is just notation though) and each neuron has its own set of weights to match the number of features.

2. Having read this question and its answers, I understand that stacking layers just adds more non-linearity. While stacking a linear function on a linear function, our function is still linear. But there is a limit to how much we can learn at any order, and adding layers just means we can have a more complex decision boundary. That said, a model too deep can be prone to overfitting.

Batch Normalisation then aims to normalise inputs to make distributions simpler for each layer while also making backpropagation faster.

Dropout looks to prevent overfitting through randomly dropping neurons to see if we can learn without that specific layer.

But why would there be multiple neurons per layer? Is it just like poling/averaging where you have more candidates to chose from to ensure confidence and correctness? Or is each neuron learning something else entirely?

Answer 1

Or is each neuron learning something else entirely?

This is the purpose, although I wouldn't say they learn entirely different things since they might have some correlation. Each neuron will have its own view of the data and produces outputs according to it. It's like multiple people from different perspectives looking at the same thing, sharing their opinions, and these opinions are aggregated over and over again in the subsequent layers.

Answer 2

Neural network is basically a tensor function (tensors = scalars, vectors, matrices etc.). It takes stuff in (e.g. vector of some features), compoundly apply some operations like matrix multipication or activation functions, and produces stuff out (e.g. vector of probability distribution over some classes).

The term neuron basically refers to a particular element of a tensor. Having 1 neuron in a layer = the layer would consist of a scalar value (or equivalently vector of size 1, matrix of size 1x1 etc.). Usually, it's good idea to have more dimensional tensors in a network.

Example: You have network that takes an input of 10-dimensional vector, apply matrix multiplication (the matrix is a learnable parameter) to get a vector of 64-dimension, then apply elementwise activation function (e.g. RELU), then apply e.g. softmax classification to get probability distribution over 7 classes.

The hidden layer have 64 neurons, which means that the vector have a dimensionality of 64. If it would have only a dimensionality of 1, the capacity (complexity) of the network would be much lower.

Answer 3

I think I have the answer having taken some time out.

When we run a neuron, it is a glorified $z=W.T*x +b$ followed by $sigmoid(z)$

As a result, the output is just a single value in a certain range. If a layer has 100 neurons, it has 100 such features.

When we cascade and add multiple layers, the output of $L1$ is the input to $L2$. As a result, if $L1$ has only a single neuron, the next layer has only one feature to learn from. So adding more layers just allows us to get more features and better represent our data.

Answer 4

Expanding upon what @gunes said, if you take a shallow network and explicitly trace out what the network's forward pass is. You can see the adding of additional nodes (within a layer) allows the network to minimise its prediction error.

Let's assume we have some Feed-Forward Neural Network (FFNN) that takes in a single input and a single output and has a single hidden layer of N hidden nodes, and a non-linear activation function denoted, $\sigma$, and the entire forward pass of the network can be described as $\hat{y} = f(x)$, where $\hat{y}$ is the prediction of the true output $y$.

For N = 1, the network can be written as, $$ \hat{y} = w^{(2)}\sigma(w^{(1)}x_j + b^{(1)})$$ for a given number of hidden nodes, N, it's written as a sum of single neurons, i.e., $$ \hat{y} = \sum_i w^{(2)}_{i}\sigma(w^{(1)}_ix_j + b^{(1)}_i) $$

If we look at this from a linear regression standpoint we would minimise a loss function, for example the Mean Absolute Error (MAE), in order to tune the weights and biases such that our network accurately predicts our training data.

Let's apply the Universal Approximation Theorem to see what adding more neurons does. Let's say we have a single hidden node in our network and we give our network a single example pair $(x_j, y_j)$, this would give a prediction $\hat{y_j}$ which would have an error of $\vert y_j - \hat{y}_j \vert$. We would then use some optimization algorithm (SGD, RMSprop, Adam etc...) to minimise this error. However, a single hidden node can only 'learn' so much and will eventually hit a local minimum where it'll learn no more.

This would be written out like,

$$\hat{y}_j = w^{(2)}_1\sigma(w^{(1)}_1 x_j + b^{(1)}_1)$$

and let's say (for the sake of argument) the lowest error we can get is 10. This error of 10 can be minimising by adding a new neuron. Let's say the target $y_j$ is 20 and our model predicts $\hat{y}_j$ = 10 (giving the error of 10 from above). We can add a new neuron to try and reduce this error by getting the new neuron to output a contribution to the total sum which minimising our error. And, let's say the best the 2nd neuron can do is an output of 5 - which would bring out total MAE to 5.

$$\hat{y}_j = \underbrace{w^{(2)}_1\sigma(w^{(1)}_1 x_j + b^{(1)}_1)}_{\text{first neuron outputs 10}} + \underbrace{w^{(2)}_2\sigma(w^{(1)}_2 x_j + b^{(1)}_2)}_{\text{second neuron outputs 5}} $$

In this two neuron case, the best the model can do is 15 and in principle, we can keep adding neurons to the layer to reduce this error. Leading to the Univeral Approximation Theorem stating that a neural network can approximate any continuous function if given enough nodes to do so.

In short, the neurons for a given layer allow the network to transform the input in a certain manner and having fewer neurons restricts the ability of the layer to transform the data in the 'correct' way. Which in the case of deeper networks this restricts the ability of higher layers in transforming the data because the initial transformation they get are 'further' away (in terms of a loss metric) from the exact transformation!