What does shallow, deep, shallower, deeper, mean in the context of neural network layers?
For context:
$\begingroup$
$\endgroup$
1
-
$\begingroup$ For reference, this image is from Section 9.2 of [Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.] $\endgroup$– Jake LeviCommented Apr 12 at 14:54
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Just one comment on top of Tim's answer, "shallow" and "deeper" in the cited context are just relative descriptions. It is not referring to certain layers as shallow or deep, but stating that a certain layer $L_i$ has larger receptive field than its precedent layer $L_{i-1}$ due to the effect of convolution and pooling.
$\begingroup$
$\endgroup$
4
Shallower layers are the layers closer to input layer, while deeper layers are those more distant from input layer. However this is not a formal terminology, but rather informal, descriptive language.
-
2$\begingroup$ No I mean, a single neural network has both shallow and deep layers. I am not talking about NNs in general. Within a single NN, which layer would be considered a shallow and which one considered a deep one. Does deep mean number of nodes or deep means closer to the center or deep means closer to the final output? $\endgroup$– confusedCommented Jul 26, 2020 at 17:09
-
$\begingroup$ @confused if you want to use such distinction, than closer to input layer, would be "shallower", though I don't think that such terminology is useful for anything, better just say "input layer", "first hidden layer" etc. $\endgroup$– TimCommented Jul 26, 2020 at 17:11
-
$\begingroup$ It's used in the Deep Learning book by Goodfellow. $\endgroup$– confusedCommented Jul 26, 2020 at 17:13
-
$\begingroup$ I think Chien-Yao Want et al "YOLOR" paper gives insight and context for shallow and deep layers. It seems deep layers refer to those closer to the output. "In the general definition of neural networks, the features obtained from the shallow layers are often called explicit knowledge, and the features obtained from the deep layers are called implicit knowledge. In this paper, we call the knowledge that directly correspond to observation as explicit knowledge. As for the knowledge that is implicit in the model and has nothing to do with observation, we call it as implicit knowledge." $\endgroup$– SdlionCommented Jan 17, 2022 at 3:23
$\begingroup$
$\endgroup$
3
The notion of deep neural networks actually goes way back. At the time of McCulloch and Pitts (1943) and later with Rosenblatt's perceptron (1958), no learning algorithm was available that could train neural networks with more than one hidden layer.
In 1985, the backpropagation algorithm was developed and published by Rumelhart et al. That can train neural networks with nonlinear activation nodes (in the hidden layer). 'Backprop' even works for $k$ hidden layers. Since then, deep neural networks with more than one hidden layer have been trained and tested numerous times. During the 90-ties, neural networks lost some of their 'hype' and also many researchers began systematically to compare the performance of neural networks with that of alternative classifier and regression models. There are applications where a linear discriminant outperforms a well-trained nonlinear neural network.
After the millennium, an understanding arose that more hidden layers can add higher-order (statistical) relations to neural networks (deep NNs in image processing). Specifically for computer vision, deep NNs with multiple hidden layers were developed that turned out to learn feature detection filters in the first hidden layer. Experiments have also been done with initializing the weights of this first hidden layer using the known coefficients of linear filters (edge-detection, corner detection, T-junction detection - a complete filter bank as a matter of fact). Neural networks with pixel input is a special type of application because of the spatial arrangement of an image. Deep neural networks have proven successful on many kinds of data: image, symbolic, speech, recursive and more.
So, with deep neural networks we mean more than one hidden layer.
I suggest you to have a look at the groundbreaking paper by LeCun (LeCun, Y., Bengio, Y. Hinton, G. Deep learning. Nature 521, 436–444, 2015).
-
1$\begingroup$ Does this actually answer the question? $\endgroup$– doubllleCommented Jul 26, 2020 at 20:24
-
-
$\begingroup$ I think the question does not refers to the difference between deep and shallow NN's. But it's about deep and shallow layers inside a Deep Neural Network. As mentioned in Chien-Yao Wang et al "YOLOR" paper's introduction. $\endgroup$– SdlionCommented Jan 17, 2022 at 3:12