I wa currently reading up on standard neural network and become a bit confused in the terms used relating training deep neural network versus a normal neural network. Are they trained similarly or differently if so? what difference?
In general, a deep neural network is a neural network with many layers. In practice, there are some qualitative differences, which I'll go into in a sec, but they're still trained using back propagation, in general. As far as qualitative differences, deep networks tend to use layers which facilitate training:
- convolutional layers have fewer parameters to learn, so can be stacked up relatively deep
- skip-connections and residual networks facilitate the flow of gradients backwards through the network, so enable back-propagation through relatively large numbers of layers, up to 50-100, instead of maybe less than 10ish
- dropout facilitates finding reasonably good minimum, not getting stuck in local minimum too much
- batch normalization facilitates using high learning rates without the gradients becoming too extreme