I attempted to build a deep network (e.g. deep autoencoder) for some object classification, my result showed that the deep networks is worst than shallow network. However, from what I have read from lecture, deep network perform well. This raise me a question: does deep always better than shallow? If not, in what situation?

Is that any existing problem (published) showing that a shallow network is better than a deep network?


1 Answer 1


Deep isn't necessarily better. Deep networks have more parameters, so they are more prone to overfitting (the same way as wide networks are). Deep networks also tend to suffer more from the problem of the vanishing/exploding gradients, so they are trickier to train and you need to apply one of the "special" deep techniques to get good results (pre-training, HF, dropout + momentum, maxout, etc).

Until recently it was in fact common sense that adding more layers usually didn't help. There must be a good bunch of older papers stating this. I don't think there are many current works claiming that adding new layers didn't help, because it wouldn't be very a surprising results in most cases. It's also partially justified by the publication bias, which decreases probability of publishing works with such "negative" results.

  • $\begingroup$ good insight, thx! I also see several cases like digit classification actually gives out good result in deep network. So, its all depend on problems then. Also, i think the advance of greedy laywise-training make deep network training easier. What does HF mean? $\endgroup$ Oct 21, 2014 at 16:14
  • $\begingroup$ HF = Hessian-Free Optimization, sorry. It's good to notice that layerwise pre-training has fallen quite out of favor. Many groups advocates the use of these more advanced techniques instead. $\endgroup$ Oct 21, 2014 at 16:36
  • $\begingroup$ Cool, never heard of it, but it apparently has a lot of citations! One question, do you know if there is any good resources that I can find the most up-to-date deep learning technology? $\endgroup$ Oct 21, 2014 at 17:24
  • 1
    $\begingroup$ To find the "most up-to-date" you'll need to read the papers published in this year. It's a very active field of research and things are improving fast. Anyway, before reading the most recent stuff it's highly recommended to have a good grasp of the foundations. Searching "deep learning recent developments" will yield many good sources describing everything older than one year. Youtube and videolectures.net have also many excellent contents. $\endgroup$ Oct 21, 2014 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.