Basically the above. To clarify, I'm referring to feedforward networks (would recurrent networks technically have infinite depth). A reference would be appreciated, as I'm going to use this for sixth form work.
I think if we do not consider the computational constraints, there is no limit on the depth. In other words, we can define and build a neural network with very complex structure, but may not able to train it.
In real world, the depth will depend on your data size (if you have huge/unlimited data, say, cat pictures on Internet, you may need a very deep model.), and hardware / software (if you are using GPU, training on deep model may be much faster than CPU.), especially, number of hidden unites in the layer. Number of parameters of the model would depend on both depth and "width" of the network. Roughly speaking, a network with 10,000 layer and each layer has 2 units vs 1,000 layer, each layer has 20 unites, would have "similar" complexity.
If we consider what people are using, this link says
even beyond 1200 layers and still yield meaningful improvements in test error