I was listening to a talk and saw this slide:
How true is it?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
I was browsing the AI StackExchange and ran across a very similar question: What distinguishes “Deep Learning” from other neural networks?
Since the AI StackExchange will close tomorrow (again), I'll copy the two top answers here (user contributions licensed under cc by-sa 3.0 with attribution required):
Two well-cited 2006 papers brought the research interest back to deep learning. In "A fast learning algorithm for deep belief nets", the authors define a deep belief net as:
[...] densely-connected belief nets that have many hidden layers.
We find almost the same description for deep networks in "Greedy Layer-Wise Training of Deep Networks":
Deep multi-layer neural networks have many levels of non-linearities [...]
Then, in the survey paper "Representation Learning: A Review and New Perspectives", deep learning is used to encompass all techniques (see also this talk) and is defined as:
[...] constructing multiple levels of representation or learning a hierarchy of features.
The adjective "deep" was thus used by the authors above to highlight the use of multiple non-linear hidden layers.
Just to add to @mommi84 answer.
Deep learning is not limited to neural networks. This is more broad concept than just Hinton's DBNs etc. Deep learning is about the
constructing multiple levels of representation or learning a hierarchy of features.
So it is a name for hierarchical representation learning algorithms. There are deep models based on Hidden Markov Models, Conditional Random Fields, Support Vector Machines etc. The only common thing is, that instead of (popular in '90s) feature engineering, where researchers were trying to create set of features, which is the best for solving some classification problem - these machines can work out their own representation from raw data. In particular - applied to image recognition (raw images) they produce multi level representation consisting of pixels, then lines, then face features (if we are working with faces) like noses, eyes, and finally - generalized faces. If applied to Natural Language Processing - they construct language model, which connects words into chunks, chunks into sentences etc.
Another interesting slide:
This is certainly a question that will provoke controversy.
When neural networks are used in deep learning they are typically trained in ways that weren't used in the 1980's. In particular strategies that pretrain individual layers of the neural network to recognize features at different level are claimed to make it easier to train networks with several layers. That's certainly a new development since the 1980's.
The key is the word "deep" in deep learning. Someone (forgot ref) in the 80s proved that all non-linear functions could be approximated by a single layer neural network with, of course, a sufficiently large number of hidden units. I think this result probably discouraged people from seeking deeper network in the earlier era.
But depth of the network is what proved to be the crucial element in hierarchical representation that drives the success of many of today's applications.
Not exactly, the ANN starts in the 50s. Check out one of ML rock stars Yann LeCun's slides for an authentic and comprehensive intro. http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf