Why we tend to design more deep neural network instead of shallow?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ This question would not be a duplicate if the question was, "Why have deeper neural networks risen in popularity?" $\endgroup$– TrynnaDoStatCommented Dec 23, 2016 at 16:51
-
$\begingroup$ @TrynnaDoStat, yes, I just wanna know what is the advantage of deeper NN compared with shallow NN. $\endgroup$– GoingMyWayCommented Dec 24, 2016 at 2:07
-
$\begingroup$ This question has been edited and is no longer a duplicate. I believe it should be reopened. $\endgroup$– TrynnaDoStatCommented Dec 24, 2016 at 2:37
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
To put it simply, a deeper neural network fits to more of the nuances of the data. In many contexts, this doesn't make sense. Consider the following scatterplot,
The red line is fitting to the "nuances" whereas the blue line is generalizing the data. In many contexts, the red line is a poor fit.
However, in very low-noise situations fitting to all of the nuances data does make sense. An example of a low-noise situation is image recognition. If I show you a picture of a dog, there's little-to-no ambiguity on what is in the picture. This idea of fitting to the "nuances" used to be called "fitting to the noise" and goes against many classical statistical dogma. New(ish) problems, like image recognition, have risen in popularity and so has the need for deeper neural networks compared to shallow neural networks.
answered Dec 23, 2016 at 16:37
-
$\begingroup$ You should be able to fit the "nuances" just as well by making a wider NN. $\endgroup$ Commented Dec 23, 2016 at 22:44
-
1$\begingroup$ I'm not contrasting deep vs. wide. Rather, deep vs. shallow. I think the tendency to solve problems by going deeper is not trivial to those who've learned statistical modeling as a over vs. under fitting problem. $\endgroup$ Commented Dec 24, 2016 at 0:07