# Is training a deep neural network still referred as training using back propagation?

I wa currently reading up on standard neural network and become a bit confused in the terms used relating training deep neural network versus a normal neural network. Are they trained similarly or differently if so? what difference?

• Depends what you mean by similarly. Deep networks are still trained using backpropagation, but adding more and more layers requires sophisticated ways to combat the vanishing/exploding gradient - careful weights initialization, batch normalization, identity connections, skip identity connections, complex optimizers etc. – Łukasz Grad May 4 '17 at 21:28

In general, a deep neural network is a neural network with many layers. In practice, there are some qualitative differences, which I'll go into in a sec, but they're still trained using back propagation, in general. As far as qualitative differences, deep networks tend to use layers which facilitate training:

• convolutional layers have fewer parameters to learn, so can be stacked up relatively deep
• skip-connections and residual networks facilitate the flow of gradients backwards through the network, so enable back-propagation through relatively large numbers of layers, up to 50-100, instead of maybe less than 10ish
• dropout facilitates finding reasonably good minimum, not getting stuck in local minimum too much
• batch normalization facilitates using high learning rates without the gradients becoming too extreme