3
$\begingroup$

Practitioners know that if we increase the number of full-connected layers in Neural Network (NN), then at some points the NN performance starts to degrade.

  • The natural reason is that we have classical overfitting, as we have more parameters to estimate.
  • Among other possible reasons are problems with backpropagation of gradients through deep networks
  • etc.

My question is: are there any works that claim to found the ultimate reason for this effect and do they propose a way to fight performance degradation when working with a large number of fully-connected layers?

Both statistical-learning-based and empirical-based answers are welcome.

Improve this question
$\endgroup$
3
  • $\begingroup$ It's feasible to train a 10,000 layer CNN. If we accept this premise, then it seems that we can exclude the explanation that very deep networks cannot be trained because of problems with backprop. "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks" Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington arxiv.org/abs/1806.05393 (earlier version of my comment deleted because I misstated my meaning) $\endgroup$
    – Sycorax ♦
    Nov 21, 2019 at 13:02
  • $\begingroup$ When you say "performance degradation" are you referring to generalization (how well the model does on a holdout according to some metric) or something else? $\endgroup$
    – Sycorax ♦
    Nov 21, 2019 at 18:13
  • $\begingroup$ @sycorax-says-reinstate-monica I'm referring to test error or another quality measure. $\endgroup$
    – Alexey Zaytsev
    Dec 2, 2019 at 12:24

1 Answer 1

Reset to default
2
$\begingroup$

Those reasons you state are valid, but I see it more as a parameter efficiency thing. Compared to a CNN, a fully-connected NN needs to have a lot more parameters.

To prove my point I'll substitute a popular CNN (e.g. the ResNet-50) with a fully-connected NN. Both will be trained on the same input, e.g. $224\times224\times3$ images, (or $150,528$ inputs). Both models' outputs will be the same so I'll ignore the final layer for both.

  1. A ResNet-50 (ignoring its last layer) has $23,587,712$ parameters.
  2. A NN with a single hidden layer, would obviously depend on the size of that layer.
    • With let's say $2,048$ neurons would need $308,283,392$ parameters just for the that layer. That's more than $10$ times more than the ResNet.
    • If I wanted to make things fair and have the same number of outputs as the ResNet-50 has (which is $7\times7\times2,048=100,352$) would need $15,105,886,208‬$ parameters. THat amounts to $640$ times more parameters than the ResNet.

So, let's recap: if you wanted to swap the whole convolutional part of the 50-layer ResNet with a single FC layer, you would need $640$ times more parameters. Imagine wanting to add more than one FC layers, so that you can extract the high-level features that CNNs are known for. Image data is just too high-dimensional for fully-connected NNs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.