I'm interested in knowing what is the benefit of having 3 fully-connected layers in a Neural Network instead of 2. Many deep Neural Networks such as ImageNet do this. Why is this superior as compared to having 2 layers with the same number of parameters? (i.e., more neurons in each layer)


marked as duplicate by amoeba, kjetil b halvorsen, gung, mdewey, Dougal Apr 21 '17 at 17:46

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ If you need a detailed description you can check out "Learning Deep Architectures for AI". The 2nd chapter (Theoretical Advantages of Deep Architectures) provides some good insights about superiority of deep structures. $\endgroup$ – yasin.yazici May 16 '15 at 16:32

A lot of the benefit in deep neural networks comes from the ability of lower layers to learn representations that the higher layers can then use to perform their classification.

In a typical image recognition setting, the first layer might learn simple components of an image, like edges, the next layer might learn more complicated features that are combinations of edges, like shapes etc. and the 3rd layer can learn combinations of shapes, for example it could learn face like features if performing face detection, or it might learn components of vehicles (like wheels, headlights etc.) in a vehicle recognition task.

This image demonstrates this type of learning:

enter image description here

That is of course the ideal case. Feature learning and representation is an active research field and learning these types of features requires a lot of effort and domain knowledge.

  • $\begingroup$ Where did you get that image? It's great. $\endgroup$ – pir May 15 '15 at 13:14
  • $\begingroup$ Just to clarify. The benefit of having 3 layers over 2 would be in very hierarchically structured data, right? I guess that for some tasks 2 layers would be better. $\endgroup$ – pir May 15 '15 at 13:15
  • $\begingroup$ Yes, as with most things in ANN it all depends on your problem. The image is from this article: datarobot.com/blog/a-primer-on-deep-learning $\endgroup$ – Bar May 15 '15 at 13:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.