What is a "shallow" layer and a "deep" layer of a neural network? What does shallow, deep, shallower, deeper, mean in the context of neural network layers?
For context:

 A: Just one comment on top of Tim's answer, "shallow" and "deeper" in the cited context are just relative descriptions. It is not referring to certain layers as shallow or deep, but stating that a certain layer $L_i$ has larger receptive field than its precedent layer $L_{i-1}$ due to the effect of convolution and pooling.
A: Shallower layers are the layers closer to input layer, while deeper layers are those more distant from input layer. However this is not a formal terminology, but rather informal, descriptive language.
A: The notion of deep neural networks actually goes way back. At the time of McCulloch and Pitts (1943) and later with Rosenblatt's perceptron (1958), no learning algorithm was available that could train neural networks with more than one hidden layer.
In 1985, the backpropagation algorithm was developed and published by Rumelhart et al. That can train neural networks with nonlinear activation nodes (in the hidden layer). 'Backprop' even works for $k$ hidden layers. Since then, deep neural networks with more than one hidden layer have been trained and tested numerous times. During the 90-ties, neural networks lost some of their 'hype' and also many researchers began systematically to compare the performance of neural networks with that of alternative classifier and regression models. There are applications where a linear discriminant outperforms a well-trained nonlinear neural network.
After the millennium, an understanding arose that more hidden layers can add higher-order (statistical) relations to neural networks (deep NNs in image processing). Specifically for computer vision, deep NNs with multiple hidden layers were developed that turned out to learn feature detection filters in the first hidden layer. Experiments have also been done with initializing the weights of this first hidden layer using the known coefficients of linear filters (edge-detection, corner detection, T-junction detection - a complete filter bank as a matter of fact). Neural networks with pixel input is a special type of application because of the spatial arrangement of an image. Deep neural networks have proven successful on many kinds of data: image, symbolic, speech, recursive and more.
So, with deep neural networks we mean more than one hidden layer.
I suggest you to have a look at the groundbreaking paper by LeCun (LeCun, Y., Bengio, Y. Hinton, G. Deep learning. Nature 521, 436–444, 2015).
