Why is training of extremely deep fully-connected NNs difficult? Practitioners know that if we increase the number of full-connected layers in Neural Network (NN), then at some points the NN performance starts to degrade. 


*

*The natural reason is that we have classical overfitting, as we have more parameters to estimate. 

*Among other possible reasons are problems with backpropagation of gradients through deep networks 

*etc.


My question is: are there any works that claim to found the ultimate reason for this effect and do they propose a way to fight performance degradation when working with a large number of fully-connected layers?
Both statistical-learning-based and empirical-based answers are welcome.
 A: Those reasons you state are valid, but I see it more as a parameter efficiency thing. Compared to a CNN, a fully-connected NN needs to have a lot more parameters.
To prove my point I'll substitute a popular CNN (e.g. the ResNet-50) with a fully-connected NN. Both will be trained on the same input, e.g. $224\times224\times3$ images, (or $150,528$ inputs). Both models' outputs will be the same so I'll ignore the final layer for both. 


*

*A ResNet-50 (ignoring its last layer) has $23,587,712$ parameters.

*A NN with a single hidden layer, would obviously depend on the size of that layer.


*

*With let's say $2,048$ neurons would need $308,283,392$ parameters just for the that layer. That's more than $10$ times more than the ResNet.

*If I wanted to make things fair and have the same number of outputs as the ResNet-50 has (which is $7\times7\times2,048=100,352$) would need $15,105,886,208‬$ parameters. THat amounts to $640$ times more parameters than the ResNet.



So, let's recap: if you wanted to swap the whole convolutional part of the 50-layer ResNet with a single FC layer, you would need $640$ times more parameters. Imagine wanting to add more than one FC layers, so that you can extract the high-level features that CNNs are known for. Image data is just too high-dimensional for fully-connected NNs. 
