Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I was listening to a talk and saw this slide:

enter image description here

How true is it?

share|improve this question
I need more context. –  cardinal Feb 16 '14 at 23:11
it would help if you cited the researcher. to me, at its core deep learning involves much larger networks in neuron counts and more layers. admittedly that is somewhat implied by the above points which seem roughly accurate. the above points facilitate larger networks. –  vzn Aug 15 '14 at 16:34

3 Answers 3

This is certainly a question that will provoke controversy.

When neural networks are used in deep learning they are typically trained in ways that weren't used in the 1980's. In particular strategies that pretrain individual layers of the neural network to recognize features at different level are claimed to make it easier to train networks with several layers. That's certainly a new development since the 1980's.

share|improve this answer

I was browsing the AI StackExchange and ran across a very similar question: What distinguishes “Deep Learning” from other neural networks?

Since the AI StackExchange will close tomorrow (again), I'll copy the two top answers here (user contributions licensed under cc by-sa 3.0 with attribution required):

Author: mommi84less

Two well-cited 2006 papers brought the research interest back to deep learning. In "A fast learning algorithm for deep belief nets", the authors define a deep belief net as:

[...] densely-connected belief nets that have many hidden layers.

We find almost the same description for deep networks in "Greedy Layer-Wise Training of Deep Networks":

Deep multi-layer neural networks have many levels of non-linearities [...]

Then, in the survey paper "Representation Learning: A Review and New Perspectives", deep learning is used to encompass all techniques (see also this talk) and is defined as:

[...] constructing multiple levels of representation or learning a hierarchy of features.

The adjective "deep" was thus used by the authors above to highlight the use of multiple non-linear hidden layers.

Author: lejlot

Just to add to @mommi84 answer.

Deep learning is not limited to neural networks. This is more broad concept than just Hinton's DBNs etc. Deep learning is about the

constructing multiple levels of representation or learning a hierarchy of features.

So it is a name for hierarchical representation learning algorithms. There are deep models based on Hidden Markov Models, Conditional Random Fields, Support Vector Machines etc. The only common thing is, that instead of (popular in '90s) feature engineering, where reserachers were trying to create set of features, which is the best for solving some classificaiton problem - these machines can work out their own representation from raw data. In particular - applied to image recognition (raw images) they produce multi level representation consisting of pixels, then lines, then face features (if we are working with faces) like noses, eyes, and finally - genralized faces. If applied to Natural Language Processing - they construct language model, which connects words into chunks, chunks into sentences etc.

share|improve this answer

Dropout, from Hinton in 2006, is said to be the greatest improvement in deep learning of the last 10 years, because it reduces a lot overfitting.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.