Is there academic research on how to choose the number of layers and neurons in a neural network? I did some research and I understand that there is no right answer in choosing the number of layers and neurons in a neural network architecture. I have seen some guidelines and rules of thumb in posts and blogs. Even though they are helpful, I am wondering if there is any academic paper on them.
Do you have any study/work/research about how to choose the number of layers and neurons to share?
 A: Often more neurons/layers than needed are chosen, and then regularization techniques are used to get a lower effective complexity than the potential complexity. But even then, there is not a canonical answer to how many neurons/layers to choose.
Source: Xia Hu et al. "Model Complexity of Deep Learning: A Survey." (2021).
A: The research field investigating this is called Neural Architecture Search. A recent very detailed review could be a great guide :
Automated Deep Learning: Neural Architecture Search Is Not the End.
Xuanyi Dong, David Jacob Kedziora, Katarzyna Musial, Bogdan Gabrys.
arXiv:2112.09245 (2021)
A: There are empirical observations that can guide you in the number of neurons (so go read papers and classical examples), but in general it's just testing what works the best with the least number of neurons. Some techniques like Pruning start with a big network and try to reduce the number of useful neurons while keeping the same performance.
On the other hand there are computational reasons to choose certain number of neurons, like power of 2. This is why you commonly see layers of 64,128,256 etc neurons and same for convolutional filters. Indeed, using power of 2 can enable more efficient algorithm and also exploits some properties of your GPU like Tensor Cores that can only be used when sizes of layers are multiples of 8. It's also more convenient when you have an "encoder-decoder" structure.
