Why we tend to design more deep neural network instead of shallow?

  • $\begingroup$ This question would not be a duplicate if the question was, "Why have deeper neural networks risen in popularity?" $\endgroup$ – TrynnaDoStat Dec 23 '16 at 16:51
  • $\begingroup$ @TrynnaDoStat, yes, I just wanna know what is the advantage of deeper NN compared with shallow NN. $\endgroup$ – GoingMyWay Dec 24 '16 at 2:07
  • $\begingroup$ This question has been edited and is no longer a duplicate. I believe it should be reopened. $\endgroup$ – TrynnaDoStat Dec 24 '16 at 2:37

To put it simply, a deeper neural network fits to more of the nuances of the data. In many contexts, this doesn't make sense. Consider the following scatterplot,

enter image description here

The red line is fitting to the "nuances" whereas the blue line is generalizing the data. In many contexts, the red line is a poor fit.

However, in very low-noise situations fitting to all of the nuances data does make sense. An example of a low-noise situation is image recognition. If I show you a picture of a dog, there's little-to-no ambiguity on what is in the picture. This idea of fitting to the "nuances" used to be called "fitting to the noise" and goes against many classical statistical dogma. New(ish) problems, like image recognition, have risen in popularity and so has the need for deeper neural networks compared to shallow neural networks.

  • $\begingroup$ You should be able to fit the "nuances" just as well by making a wider NN. $\endgroup$ – gung - Reinstate Monica Dec 23 '16 at 22:44
  • 1
    $\begingroup$ I'm not contrasting deep vs. wide. Rather, deep vs. shallow. I think the tendency to solve problems by going deeper is not trivial to those who've learned statistical modeling as a over vs. under fitting problem. $\endgroup$ – TrynnaDoStat Dec 24 '16 at 0:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.