1
$\begingroup$

I wa currently reading up on standard neural network and become a bit confused in the terms used relating training deep neural network versus a normal neural network. Are they trained similarly or differently if so? what difference?

$\endgroup$
  • 1
    $\begingroup$ Depends what you mean by similarly. Deep networks are still trained using backpropagation, but adding more and more layers requires sophisticated ways to combat the vanishing/exploding gradient - careful weights initialization, batch normalization, identity connections, skip identity connections, complex optimizers etc. $\endgroup$ – Łukasz Grad May 4 '17 at 21:28
1
$\begingroup$

In general, a deep neural network is a neural network with many layers. In practice, there are some qualitative differences, which I'll go into in a sec, but they're still trained using back propagation, in general. As far as qualitative differences, deep networks tend to use layers which facilitate training:

  • convolutional layers have fewer parameters to learn, so can be stacked up relatively deep
  • skip-connections and residual networks facilitate the flow of gradients backwards through the network, so enable back-propagation through relatively large numbers of layers, up to 50-100, instead of maybe less than 10ish
  • dropout facilitates finding reasonably good minimum, not getting stuck in local minimum too much
  • batch normalization facilitates using high learning rates without the gradients becoming too extreme
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.