# Why convolutional neural networks belong to deep learning?

In my idea, deep learning is a process of feature extraction.

Just like multiple layer neural networks (NN): Input1 => L1 => L2 => ... => Ln => Output1. The special aspect of deep learning is to let Output1 equal to Input1. As a result, we can get the error of Output1. Then, we can try to use backpropagation (BP) to train our model to minimize the error. When it is complete, all layers' output is the internal feature representations from edge to partial of object to full object. This made deep learning so fancy. This concept is illustrated by this picture:

Now, back to convolutional neural networks (CNN). CNN use convolution to extract features and try to learn all filters by BP. I do not see CNN generate the output similar to the input pictures. It is just convolution and pooling and so on to become very small pixel fractions, called basis.

How CNN use deep learning concept in its implementation? Why BP can train CNN model to the correct internal features of all layers?

• I took the liberty to edit the question. I assumed that by BP you meant backpropagation. – Aleksandr Blekh Apr 15 '15 at 2:37
• Can you provide a link to the source of the picture, please? It would be interesting to read discussion around it. – Vladislavs Dovgalecs Aug 20 '15 at 17:31
• It's not easy to find - Lee et al., Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations – Petr Baudis Dec 14 '15 at 19:20

First, mind that deep learning is a buzz term. There is not even a consensus of a formal definition in the research community. A discussion of the term does not lead anywhere, really. It's just a word.

That being said, convolutional nets are deep because they rely on multiple layers of feature extraction, as you said. They extract features from the input to predict an outcome.

What you refer to is a "generative" approach, i.e. the features are used to create the observation (a picture, not a class label). That is what made deep learning popular, but it is in no way limited to that.

• First, thank you for hint me that deep learning is not the same as using the concept like "Input1 => L1 => L2 => ... => Ln => Output1" and let Output1 equal to Input1 to extract layers of abstract features which hard to model using math by human. – sam Apr 15 '15 at 14:56
• Second, sorry for my unclear describing. I'm not trying to focus on the term "deep learning" definition discussion. Let me suppose the learning process "Input1 => L1 => L2 => ... => Ln => Output1" and let Output1 equal to Input1 named "ABC learning". – sam Apr 15 '15 at 15:10
• When the first time I see "ABC learning", I'm on fire with this marvelous idea. If the tuning process of "ABC learning" to be succeeded, we can get the correct representation of all layers of object concept as depicted on above picture. But I can not find where CNN trying to implement "ABC learning". – sam Apr 15 '15 at 15:10
• If CNN does not implement it, when CNN extract all basis of features, how to measure its error and then learn by BP? Is there any error in my thinking? Thank you very much~ – sam Apr 15 '15 at 15:10
• In my thinking, CNN is just implement the first part of "ABC learning". It seems lack of the second part of generative approach to become the original picture to calculate errors and then run BP to learn all features. Any ideas or suggestions are welcome~^^ – sam Apr 15 '15 at 15:26

Deep learning is a generic term that refers to the fact that a deep neural network has at least one hidden layer.

• a network would need more than one hidden layer to be a deep network, networks with one or two hidden layers are traditional neural networks, and there is no particular difficulty in training them with back-propagation (as the error signals have not become too diffuse by that stage). – Dikran Marsupial Apr 15 '15 at 11:34
• In my own thinking, deep is not related to the number of layers, but it talks about how hard the feature to be discovered. For example, we are very hard to describe what is cat. That is, the concept of cat is very deep. If we can use a structure to discover it by machine learning, I called it deep learning. Is this concept reasonable? – sam Apr 15 '15 at 16:03
• sam, your point of view makes sense to me. However, again, in my opinion "deep" refers to the architecture rather than a "deep" concept. That's why in the literature you usually find names such as deep neural network (DNN), deep belief network (DBN), and so on. – njk Apr 15 '15 at 18:54
• if neural networks with one hidden layer are considered "deep" then there is no distinction between "neural networks" and "deep neural networks" (i.e. the "deep" part conveys no information). A single layer of neurons doesn't really constitute a neural network as each neuron acts independently of the others. – Dikran Marsupial Apr 16 '15 at 9:29
• In the following video (minute 1:20) Geoffrey Hinton suggests that a neural network with more that one hidden layer can be called "deep". youtube.com/… – njk Jul 2 '15 at 22:34

The deep learning is an approach where you have a lot of relatively simple layers. You increase learning capabilities by increasing the number of layers, as opposed to increased complexity of layers. You could for instance come up with very fancy output functions, maybe nonlinear functions of inputs or complicated connections. Instead you stick with simple things like ReLU and liner combination and softmax, but stack a lot of layers one on top of other. That's why CNN perfectly fits into this very generic and rather vague definition of deep learning. Look at CNN's components, they are usually very simple MAX, convolutions etc.