At what point do we start classifying multi layered neural networks as deep neural networks or to put it in another way 'What is the minimum number of layers in a deep neural network?'
One of the earliest deep neural networks has three densely connected hidden layers (Hinton et al. (2006)).
In 2014 the "very deep" VGG netowrks Simonyan et al. (2014) consist of 16+ hidden layers.
In 2016 the "extremely deep" residual networks He et al. (2016) consist of 50 up to 1,000+ hidden layers.
As per the literature,
Schmidhuber, J. (2015). "Deep Learning in Neural Networks: An Overview". Neural Networks. 61: 85–117. arXiv:1404.7828free to read. doi:10.1016/j.neunet.2014.09.003.
It is said that
There is no universally agreed upon threshold of depth dividing shallow learning from deep learning, but most researchers in the field agree that deep learning has multiple nonlinear layers (CAP > 2) and Schmidhuber considers CAP > 10 to be very deep learning
A chain of transformations from input to output is a Credit Assignment Path or CAP. For a feedforward neural network, the depth of the CAPs, and thus the depth of the network, is the number of hidden layers plus one.