# Where and why does deep learning shine?

With all the media talk and hype about deep learning these days, I read some elementary stuff about it. I just found that it is just another machine learning method to learn patterns from data. But my question is: where does and why this method shine? Why all the talk about it right now? I.e. what is the fuss all about?

• Look at Geoff Hinton's and Andrew Ng's qualitative youtube lectures for an easy overview of why it's so good. – Jase Feb 17 '14 at 13:56

The main purported benefits:

(1) Don't need to hand engineer features for non-linear learning problems (save time and scalable to the future, since hand engineering is seen by some as a short-term band-aid)

(2) The learnt features are sometimes better than the best hand-engineered features, and can be so complex (computer vision - e.g. face-like features) that it would take way too much human time to engineer.

(3) Can use unlabeled data to pre-train the network. Suppose we have 1000000 unlabeled images and 1000 labeled images. We can now drastically improve a supervised learning algorithm by pre-training on the 1000000 unlabeled images with deep learning. In addition, in some domains we have so much unlabeled data but labeled data is hard to find. An algorithm that can use this unlabeled data to improve classification is valuable.

(4) Empirically, smashed many benchmarks that were only seeing incremental improvements until the introduction of deep learning methods.

(5) Same algorithm works in multiple areas with raw (perhaps with minor pre-processing) inputs.

(6) Keeps improving as more data is fed to the network (assuming stationary distributions etc).

Another important point in addition to the above (I don't have sufficient rep to merely add it as a comment) is that it is a generative model (Deep Belief Nets at least) and thus you can sample from the learned distributions - this can have some major benefits in certain applications where you want to generate synthetic data corresponding to the learned classes/clusters.

• This is not a general property of deep learning, but of the concrete model in question. E.g. you can sample from a Gaussian but not from a logistic regression. You can also sample from a variety of deep learning models, e.g. Deep belief nets as you said, deep Boltzmann machines, Deep latent Gaussian models, etc. But you cannot sample from drednets, which are the models that have been used in all the impressive applications. – bayerj Sep 22 '14 at 10:36
• ... Why can't you sample from a logistic regression? – Hong Ooi Sep 22 '14 at 10:46
• Given an LR model $p(c|x)$, you can sample the class conditioned on the input. But you cannot sample an input. Thus, it is more of a discriminativ/generative distinction. – bayerj Sep 22 '14 at 11:30
• But the same holds for a gaussian regression model, if by that you mean basic linear regression. – Hong Ooi Sep 22 '14 at 11:52