Minimum number of layers in a deep neural network At what point do we start classifying multi layered neural networks as deep neural networks or to put it in another way 'What is the minimum number of layers in a deep neural network?'
 A: "Deep" is a marketing term: you can therefore use it whenever you need to market your multi-layered neural network.
A: "Deep"
One of the earliest deep neural networks has three densely connected hidden layers (Hinton et al. (2006)).

"Very Deep"
In 2014 the "very deep" VGG netowrks Simonyan et al. (2014) consist of 16+ hidden layers.

"Extremely Deep"
In 2016 the "extremely deep" residual networks He et al. (2016) consist of 50 up to 1,000+ hidden layers.

A: As per the literature,
Schmidhuber, J. (2015). "Deep Learning in Neural Networks: An Overview". Neural Networks. 61: 85–117. arXiv:1404.7828free to read. doi:10.1016/j.neunet.2014.09.003.
https://en.wikipedia.org/wiki/Deep_learning
It is said that 

There is no universally agreed upon threshold of depth dividing
  shallow learning from deep learning, but most researchers in the field
  agree that deep learning has multiple nonlinear layers (CAP > 2) and
  Schmidhuber considers CAP > 10 to be very deep learning

A chain of transformations from input to output is a Credit Assignment Path or CAP. For a feedforward neural network, the depth of the CAPs, and thus the depth of the network, is the number of hidden layers plus one. 
