I started a deep learning course (introductory one). But I have two years experience working with ML projects on my own time. This course is just for me to understand deep learning at a more fundamental level.
Starting with a single layer perceptron, it makes sense. We have a linear function to act as a hypothesis. Then that output is fed to an activation function that adds non-linearity and normalises outputs.
That totally makes sense to me. Now my understanding so far:
When we have multiple neurons in a layer, the connections to the neuron represents some weight. The neuron itself is just an activation function. (That is just notation though) and each neuron has its own set of weights to match the number of features.
Having read this question and its answers, I understand that stacking layers just adds more non-linearity. While stacking a linear function on a linear function, our function is still linear. But there is a limit to how much we can learn at any order, and adding layers just means we can have a more complex decision boundary. That said, a model too deep can be prone to overfitting.
Batch Normalisation then aims to normalise inputs to make distributions simpler for each layer while also making backpropagation faster.
Dropout looks to prevent overfitting through randomly dropping neurons to see if we can learn without that specific layer.
But why would there be multiple neurons per layer? Is it just like poling/averaging where you have more candidates to chose from to ensure confidence and correctness? Or is each neuron learning something else entirely?