What does shallow, deep, shallower, deeper, mean in the context of neural network layers?

For context: enter image description here


Just one comment on top of Tim's answer, "shallow" and "deeper" in the cited context are just relative descriptions. It is not referring to certain layers as shallow or deep, but stating that a certain layer $L_i$ has larger receptive field than its precedent layer $L_{i-1}$ due to the effect of convolution and pooling.


Shallower layers are the layers closer to input layer, while deeper layers are those more distant from input layer. However this is not a formal terminology, but rather informal, descriptive language.

  • 2
    $\begingroup$ No I mean, a single neural network has both shallow and deep layers. I am not talking about NNs in general. Within a single NN, which layer would be considered a shallow and which one considered a deep one. Does deep mean number of nodes or deep means closer to the center or deep means closer to the final output? $\endgroup$
    – confused
    Jul 26 '20 at 17:09
  • $\begingroup$ @confused if you want to use such distinction, than closer to input layer, would be "shallower", though I don't think that such terminology is useful for anything, better just say "input layer", "first hidden layer" etc. $\endgroup$
    – Tim
    Jul 26 '20 at 17:11
  • $\begingroup$ It's used in the Deep Learning book by Goodfellow. $\endgroup$
    – confused
    Jul 26 '20 at 17:13
  • $\begingroup$ I think Chien-Yao Want et al "YOLOR" paper gives insight and context for shallow and deep layers. It seems deep layers refer to those closer to the output. "In the general definition of neural networks, the features obtained from the shallow layers are often called explicit knowledge, and the features obtained from the deep layers are called implicit knowledge. In this paper, we call the knowledge that directly correspond to observation as explicit knowledge. As for the knowledge that is implicit in the model and has nothing to do with observation, we call it as implicit knowledge." $\endgroup$
    – Sdlion
    Jan 17 at 3:23

The notion of deep neural networks actually goes way back. At the time of McCulloch and Pitts (1943) and later with Rosenblatt's perceptron (1958), no learning algorithm was available that could train neural networks with more than one hidden layer.

In 1985, the backpropagation algorithm was developed and published by Rumelhart et al. That can train neural networks with nonlinear activation nodes (in the hidden layer). 'Backprop' even works for $k$ hidden layers. Since then, deep neural networks with more than one hidden layer have been trained and tested numerous times. During the 90-ties, neural networks lost some of their 'hype' and also many researchers began systematically to compare the performance of neural networks with that of alternative classifier and regression models. There are applications where a linear discriminant outperforms a well-trained nonlinear neural network.

After the millennium, an understanding arose that more hidden layers can add higher-order (statistical) relations to neural networks (deep NNs in image processing). Specifically for computer vision, deep NNs with multiple hidden layers were developed that turned out to learn feature detection filters in the first hidden layer. Experiments have also been done with initializing the weights of this first hidden layer using the known coefficients of linear filters (edge-detection, corner detection, T-junction detection - a complete filter bank as a matter of fact). Neural networks with pixel input is a special type of application because of the spatial arrangement of an image. Deep neural networks have proven successful on many kinds of data: image, symbolic, speech, recursive and more.

So, with deep neural networks we mean more than one hidden layer.

I suggest you to have a look at the groundbreaking paper by LeCun (LeCun, Y., Bengio, Y. Hinton, G. Deep learning. Nature 521, 436–444, 2015).

  • 1
    $\begingroup$ Does this actually answer the question? $\endgroup$
    – doubllle
    Jul 26 '20 at 20:24
  • $\begingroup$ It does now - thanks ! $\endgroup$ Jul 26 '20 at 20:41
  • $\begingroup$ I think the question does not refers to the difference between deep and shallow NN's. But it's about deep and shallow layers inside a Deep Neural Network. As mentioned in Chien-Yao Wang et al "YOLOR" paper's introduction. $\endgroup$
    – Sdlion
    Jan 17 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.