7
$\begingroup$

In my idea, deep learning is a process of feature extraction.

Just like multiple layer neural networks (NN): Input1 => L1 => L2 => ... => Ln => Output1. The special aspect of deep learning is to let Output1 equal to Input1. As a result, we can get the error of Output1. Then, we can try to use backpropagation (BP) to train our model to minimize the error. When it is complete, all layers' output is the internal feature representations from edge to partial of object to full object. This made deep learning so fancy. This concept is illustrated by this picture:

enter image description here

Now, back to convolutional neural networks (CNN). CNN use convolution to extract features and try to learn all filters by BP. I do not see CNN generate the output similar to the input pictures. It is just convolution and pooling and so on to become very small pixel fractions, called basis.

How CNN use deep learning concept in its implementation? Why BP can train CNN model to the correct internal features of all layers?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I took the liberty to edit the question. I assumed that by BP you meant backpropagation. $\endgroup$
    – Aleksandr Blekh
    Commented Apr 15, 2015 at 2:37
  • 1
    $\begingroup$ Can you provide a link to the source of the picture, please? It would be interesting to read discussion around it. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Aug 20, 2015 at 17:31
  • 2
    $\begingroup$ It's not easy to find - Lee et al., Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations $\endgroup$
    – Petr Baudis
    Commented Dec 14, 2015 at 19:20

4 Answers 4

Reset to default
12
$\begingroup$

First, mind that deep learning is a buzz term. There is not even a consensus of a formal definition in the research community. A discussion of the term does not lead anywhere, really. It's just a word.

That being said, convolutional nets are deep because they rely on multiple layers of feature extraction, as you said. They extract features from the input to predict an outcome.

What you refer to is a "generative" approach, i.e. the features are used to create the observation (a picture, not a class label). That is what made deep learning popular, but it is in no way limited to that.

Improve this answer
$\endgroup$
8
  • $\begingroup$ First, thank you for hint me that deep learning is not the same as using the concept like "Input1 => L1 => L2 => ... => Ln => Output1" and let Output1 equal to Input1 to extract layers of abstract features which hard to model using math by human. $\endgroup$
    – sam
    Commented Apr 15, 2015 at 14:56
  • $\begingroup$ Second, sorry for my unclear describing. I'm not trying to focus on the term "deep learning" definition discussion. Let me suppose the learning process "Input1 => L1 => L2 => ... => Ln => Output1" and let Output1 equal to Input1 named "ABC learning". $\endgroup$
    – sam
    Commented Apr 15, 2015 at 15:10
  • $\begingroup$ When the first time I see "ABC learning", I'm on fire with this marvelous idea. If the tuning process of "ABC learning" to be succeeded, we can get the correct representation of all layers of object concept as depicted on above picture. But I can not find where CNN trying to implement "ABC learning". $\endgroup$
    – sam
    Commented Apr 15, 2015 at 15:10
  • $\begingroup$ If CNN does not implement it, when CNN extract all basis of features, how to measure its error and then learn by BP? Is there any error in my thinking? Thank you very much~ $\endgroup$
    – sam
    Commented Apr 15, 2015 at 15:10
  • $\begingroup$ In my thinking, CNN is just implement the first part of "ABC learning". It seems lack of the second part of generative approach to become the original picture to calculate errors and then run BP to learn all features. Any ideas or suggestions are welcome~^^ $\endgroup$
    – sam
    Commented Apr 15, 2015 at 15:26
3
$\begingroup$

The deep learning is an approach where you have a lot of relatively simple layers. You increase learning capabilities by increasing the number of layers, as opposed to increased complexity of layers. You could for instance come up with very fancy output functions, maybe nonlinear functions of inputs or complicated connections. Instead you stick with simple things like ReLU and liner combination and softmax, but stack a lot of layers one on top of other. That's why CNN perfectly fits into this very generic and rather vague definition of deep learning. Look at CNN's components, they are usually very simple MAX, convolutions etc.

Improve this answer
$\endgroup$
2
$\begingroup$

Deep learning is a generic term that refers to the fact that a deep neural network has at least one hidden layer.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ a network would need more than one hidden layer to be a deep network, networks with one or two hidden layers are traditional neural networks, and there is no particular difficulty in training them with back-propagation (as the error signals have not become too diffuse by that stage). $\endgroup$
    – Dikran Marsupial
    Commented Apr 15, 2015 at 11:34
  • 1
    $\begingroup$ In my own thinking, deep is not related to the number of layers, but it talks about how hard the feature to be discovered. For example, we are very hard to describe what is cat. That is, the concept of cat is very deep. If we can use a structure to discover it by machine learning, I called it deep learning. Is this concept reasonable? $\endgroup$
    – sam
    Commented Apr 15, 2015 at 16:03
  • 2
    $\begingroup$ sam, your point of view makes sense to me. However, again, in my opinion "deep" refers to the architecture rather than a "deep" concept. That's why in the literature you usually find names such as deep neural network (DNN), deep belief network (DBN), and so on. $\endgroup$
    – njk
    Commented Apr 15, 2015 at 18:54
  • 2
    $\begingroup$ if neural networks with one hidden layer are considered "deep" then there is no distinction between "neural networks" and "deep neural networks" (i.e. the "deep" part conveys no information). A single layer of neurons doesn't really constitute a neural network as each neuron acts independently of the others. $\endgroup$
    – Dikran Marsupial
    Commented Apr 16, 2015 at 9:29
  • 1
    $\begingroup$ In the following video (minute 1:20) Geoffrey Hinton suggests that a neural network with more that one hidden layer can be called "deep". youtube.com/… $\endgroup$
    – njk
    Commented Jul 2, 2015 at 22:34
1
$\begingroup$

Very late but I think the real answer is more historical. Historically, deep learning refers to networks that use backpropagation in contrast to a lot of other types of neural network (Kohonen, mono-layer perceptron, oscillatory network, chaotic network, etc., etc.) that are a lot of research branches. When deep learning appears, before being the well-known hype terms used today, it was just a branch among the other and its specificity was to use backpropagation (in fact, it is its only specificity). When CNN appears, due to the fact that it also uses backpropagation, it was seen as an extension of the deep learning with flavours. The fact that there are in the same family can also be seen through the fact that it is common to meld both methods.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.