I wa currently reading up on standard neural network and become a bit confused in the terms used relating training deep neural network versus a normal neural network. Are they trained similarly or differently if so? what difference?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Depends what you mean by similarly. Deep networks are still trained using backpropagation, but adding more and more layers requires sophisticated ways to combat the vanishing/exploding gradient - careful weights initialization, batch normalization, identity connections, skip identity connections, complex optimizers etc. $\endgroup$– Łukasz GradCommented May 4, 2017 at 21:28
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In general, a deep neural network is a neural network with many layers. In practice, there are some qualitative differences, which I'll go into in a sec, but they're still trained using back propagation, in general. As far as qualitative differences, deep networks tend to use layers which facilitate training:
- convolutional layers have fewer parameters to learn, so can be stacked up relatively deep
- skip-connections and residual networks facilitate the flow of gradients backwards through the network, so enable back-propagation through relatively large numbers of layers, up to 50-100, instead of maybe less than 10ish
- dropout facilitates finding reasonably good minimum, not getting stuck in local minimum too much
- batch normalization facilitates using high learning rates without the gradients becoming too extreme
answered May 4, 2017 at 23:01