I was listening to a talk and saw this slide:

enter image description here

How true is it?

  • 6
    $\begingroup$ I need more context. $\endgroup$ – cardinal Feb 16 '14 at 23:11
  • 2
    $\begingroup$ it would help if you cited the researcher. to me, at its core deep learning involves much larger networks in neuron counts and more layers. admittedly that is somewhat implied by the above points which seem roughly accurate. the above points facilitate larger networks. $\endgroup$ – vzn Aug 15 '14 at 16:34
  • $\begingroup$ What is the source for this? $\endgroup$ – MachineEpsilon Jan 25 '18 at 13:04

I was browsing the AI StackExchange and ran across a very similar question: What distinguishes “Deep Learning” from other neural networks?

Since the AI StackExchange will close tomorrow (again), I'll copy the two top answers here (user contributions licensed under cc by-sa 3.0 with attribution required):

Author: mommi84less

Two well-cited 2006 papers brought the research interest back to deep learning. In "A fast learning algorithm for deep belief nets", the authors define a deep belief net as:

[...] densely-connected belief nets that have many hidden layers.

We find almost the same description for deep networks in "Greedy Layer-Wise Training of Deep Networks":

Deep multi-layer neural networks have many levels of non-linearities [...]

Then, in the survey paper "Representation Learning: A Review and New Perspectives", deep learning is used to encompass all techniques (see also this talk) and is defined as:

[...] constructing multiple levels of representation or learning a hierarchy of features.

The adjective "deep" was thus used by the authors above to highlight the use of multiple non-linear hidden layers.

Author: lejlot

Just to add to @mommi84 answer.

Deep learning is not limited to neural networks. This is more broad concept than just Hinton's DBNs etc. Deep learning is about the

constructing multiple levels of representation or learning a hierarchy of features.

So it is a name for hierarchical representation learning algorithms. There are deep models based on Hidden Markov Models, Conditional Random Fields, Support Vector Machines etc. The only common thing is, that instead of (popular in '90s) feature engineering, where researchers were trying to create set of features, which is the best for solving some classification problem - these machines can work out their own representation from raw data. In particular - applied to image recognition (raw images) they produce multi level representation consisting of pixels, then lines, then face features (if we are working with faces) like noses, eyes, and finally - generalized faces. If applied to Natural Language Processing - they construct language model, which connects words into chunks, chunks into sentences etc.

Another interesting slide:

enter image description here



Dropout, from Hinton in 2006, is said to be the greatest improvement in deep learning of the last 10 years, because it reduces a lot overfitting.


This is certainly a question that will provoke controversy.

When neural networks are used in deep learning they are typically trained in ways that weren't used in the 1980's. In particular strategies that pretrain individual layers of the neural network to recognize features at different level are claimed to make it easier to train networks with several layers. That's certainly a new development since the 1980's.


The key is the word "deep" in deep learning. Someone (forgot ref) in the 80s proved that all non-linear functions could be approximated by a single layer neural network with, of course, a sufficiently large number of hidden units. I think this result probably discouraged people from seeking deeper network in the earlier era.

But depth of the network is what proved to be the crucial element in hierarchical representation that drives the success of many of today's applications.


Not exactly, the ANN starts in the 50s. Check out one of ML rock stars Yann LeCun's slides for an authentic and comprehensive intro. http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf

  • $\begingroup$ This is to short for an answer by our standards, more of a comment. You should really expand it into a full answer by giving some of the important points from that slide! $\endgroup$ – kjetil b halvorsen Sep 11 '16 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.