Why we tend to design more deep neural network instead of shallow?
To put it simply, a deeper neural network fits to more of the nuances of the data. In many contexts, this doesn't make sense. Consider the following scatterplot,
The red line is fitting to the "nuances" whereas the blue line is generalizing the data. In many contexts, the red line is a poor fit.
However, in very low-noise situations fitting to all of the nuances data does make sense. An example of a low-noise situation is image recognition. If I show you a picture of a dog, there's little-to-no ambiguity on what is in the picture. This idea of fitting to the "nuances" used to be called "fitting to the noise" and goes against many classical statistical dogma. New(ish) problems, like image recognition, have risen in popularity and so has the need for deeper neural networks compared to shallow neural networks.