10
$\begingroup$

I have seen posts where the discussion was centered around the effect of big and small total number of neurons in a neural network, especially with respect to the potential of the network to overfit or underfit. The general idea I got is that few neurons underfit and too many will overfit, which makes sense.

Upon thinking about it a bit more, I think it makes sense to also talk about the effect of the number of neurons per layer. My intuition tells me that even if the number of neurons in a Deep Neural Network is "the right amount" (this being problem/model-specific), if the number of neurons in just 1 hidden layer is large and the number of neurons in the rest of the layers is small, then I would expect that the model would not perform well compared to a model with the same number of hidden layers and same total number of neurons.

So, the question is, in the analysis of overfitting and underfitting and performance of a deep neural network, what are the differences between a large/small total number of neurons and a large/small number of neurons per layer, respectively?

Improve this question
$\endgroup$
4
  • $\begingroup$ The situation in DL as I understand it is - theory says that deep networks can / should overfit, but they don't (empirically). From this (proceedings.mlr.press/v65/harvey17a/harvey17a.pdf) depth of a network (number of layers) increases VC dimension, so we'd might expect them to overfit more than shallow and wide networks. Again, that expectation doesn't match experiments / current best practices / rules of thumb. Also have a look at: arxiv.org/pdf/1611.03530.pdf $\endgroup$
    – bogovicj
    Commented Oct 14, 2020 at 14:10
  • $\begingroup$ @bogovicj As I understand it, at least in theory, the larger the number of fitting parameters, the more chance of overfitting. And both a very wide and very deep (or both) networks can overfit. But the question has to do with the number of neurons per layer VS the total number of neurons and how they affect overfitting. $\endgroup$
    – TheQuantumMan
    Commented Oct 14, 2020 at 14:14
  • $\begingroup$ Indeed. Are you interested in empirical findings or theoretical analysis? What I'm getting at is, if you care about theoretical analysis, I'd doubt you'd be able reach many practical conclusions from it.. (at least right now). $\endgroup$
    – bogovicj
    Commented Oct 14, 2020 at 14:35
  • $\begingroup$ I guess I want any information one can provide me with, theoretical or practical, even though I'm more interested in practical results. But with practical, I am looking for some justification of what's happening. $\endgroup$
    – TheQuantumMan
    Commented Oct 14, 2020 at 14:37

1 Answer 1

Reset to default
1
$\begingroup$

There are other important factors to consider for underfitting and overfitting discussion. They are regularization techniques. For example, L1, L2 regularization, pooling and data argumentation.

So, it is not only about number of neurons.

In recent work, people like to build a large network, at the same time, put a lot of regularizations on it. For example, it is OK to have a model that number of parameters is greater than number of data points.

For large number of neurons in one layer or large number of layers discussion:

Theoretically, MLP with one hidden layer, and the hidden layer with infinite number of neurons can approximate any functions.

In practice, especially in vision problems, people like to have more layers than large number of neurons in one layer (prefer deep but not wide).

Check this post for details.

Why are neural networks becoming deeper, but not wider?

EfficientNet paper is interesting to read for searching a better network structure.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.