Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I attempted to build a deep network (e.g. deep autoencoder) for some object classification, my result showed that the deep networks is worst than shallow network. However, from what I have read from lecture, deep network perform well. This raise me a question: does deep always better than shallow? If not, in what situation?
Is that any existing problem (published) showing that a shallow network is better than a deep network?
Deep isn't necessarily better. Deep networks have more parameters, so they are more prone to overfitting (the same way as wide networks are). Deep networks also tend to suffer more from the problem of the vanishing/exploding gradients, so they are trickier to train and you need to apply one of the "special" deep techniques to get good results (pre-training, HF, dropout + momentum, maxout, etc).
Until recently it was in fact common sense that adding more layers usually didn't help. There must be a good bunch of older papers stating this. I don't think there are many current works claiming that adding new layers didn't help, because it wouldn't be very a surprising results in most cases. It's also partially justified by the publication bias, which decreases probability of publishing works with such "negative" results.
