3
$\begingroup$

Practitioners know that if we increase the number of full-connected layers in Neural Network (NN), then at some points the NN performance starts to degrade.

  • The natural reason is that we have classical overfitting, as we have more parameters to estimate.
  • Among other possible reasons are problems with backpropagation of gradients through deep networks
  • etc.

My question is: are there any works that claim to found the ultimate reason for this effect and do they propose a way to fight performance degradation when working with a large number of fully-connected layers?

Both statistical-learning-based and empirical-based answers are welcome.

$\endgroup$
3
  • $\begingroup$ It's feasible to train a 10,000 layer CNN. If we accept this premise, then it seems that we can exclude the explanation that very deep networks cannot be trained because of problems with backprop. "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks" Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington arxiv.org/abs/1806.05393 (earlier version of my comment deleted because I misstated my meaning) $\endgroup$
    – Sycorax
    Commented Nov 21, 2019 at 13:02
  • $\begingroup$ When you say "performance degradation" are you referring to generalization (how well the model does on a holdout according to some metric) or something else? $\endgroup$
    – Sycorax
    Commented Nov 21, 2019 at 18:13
  • $\begingroup$ @sycorax-says-reinstate-monica I'm referring to test error or another quality measure. $\endgroup$ Commented Dec 2, 2019 at 12:24

1 Answer 1

2
$\begingroup$

Those reasons you state are valid, but I see it more as a parameter efficiency thing. Compared to a CNN, a fully-connected NN needs to have a lot more parameters.

To prove my point I'll substitute a popular CNN (e.g. the ResNet-50) with a fully-connected NN. Both will be trained on the same input, e.g. $224\times224\times3$ images, (or $150,528$ inputs). Both models' outputs will be the same so I'll ignore the final layer for both.

  1. A ResNet-50 (ignoring its last layer) has $23,587,712$ parameters.
  2. A NN with a single hidden layer, would obviously depend on the size of that layer.
    • With let's say $2,048$ neurons would need $308,283,392$ parameters just for the that layer. That's more than $10$ times more than the ResNet.
    • If I wanted to make things fair and have the same number of outputs as the ResNet-50 has (which is $7\times7\times2,048=100,352$) would need $15,105,886,208‬$ parameters. THat amounts to $640$ times more parameters than the ResNet.

So, let's recap: if you wanted to swap the whole convolutional part of the 50-layer ResNet with a single FC layer, you would need $640$ times more parameters. Imagine wanting to add more than one FC layers, so that you can extract the high-level features that CNNs are known for. Image data is just too high-dimensional for fully-connected NNs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.