27
$\begingroup$

I want to use deep learning in my project. I went through a couple of papers and a question occurred to me: is there any difference between convolution neural network and deep learning? Are these things the same or do they have any major differences, and which is better?

$\endgroup$
1
  • $\begingroup$ Tell me what is exact difference from deep learning and convolutional neural network I am some what confusion at these 2 topics $\endgroup$
    – Yamini
    May 4, 2019 at 7:52

5 Answers 5

38
$\begingroup$

Deep Learning is the branch of Machine Learning based on Deep Neural Networks (DNNs), meaning neural networks with at the very least 3 or 4 layers (including the input and output layers). But for some people (especially non-technical), any neural net qualifies as Deep Learning, regardless of its depth. And others consider a 10-layer neural net as shallow.

Convolutional Neural Networks (CNNs) are one of the most popular neural network architectures. They are extremely successful at image processing, but also for many other tasks (such as speech recognition, natural language processing, and more). The state of the art CNNs are pretty deep (dozens of layers at least), so they are part of Deep Learning. But you can build a shallow CNN for a simple task, in which case it's not (really) Deep Learning.

But CNNs are not alone, there are many other neural network architectures out there, including Recurrent Neural Networks (RNN), Autoencoders, Transformers, Deep Belief Nets (DBN = a stack of Restricted Boltzmann Machines, RBM), and more. They can be shallow or deep. Note: even shallow RNNs can be considered part of Deep Learning since training them requires unrolling them through time, resulting in a deep net.

$\endgroup$
10
  • $\begingroup$ @MiniQurak. Please correct me i am wrong. what I understand is CNN is just one the architectures of deep nets just like Autoencoders, Deep Belief Nets, Recurrent Neural Networks (RNN).. is it correct? $\endgroup$ Sep 14, 2016 at 8:13
  • $\begingroup$ Yes, that's correct. $\endgroup$
    – MiniQuark
    Sep 14, 2016 at 8:19
  • $\begingroup$ then can we can say which architecture is better depending on the dataset? or what are the key factors? $\endgroup$ Sep 14, 2016 at 8:32
  • 2
    $\begingroup$ CNNs are great for image recognition tasks. They also shine whenever the data has some form of hierarchical structure, with local patterns (eg. line segments) assembled into large patterns (eg. squares, triangles), assembled into larger patterns (eg. house, plane). This works well for speech recognition tasks. RNNs are great for time series (eg. predicting the weather), and generally handling sequences of data (eg. sentences). They are used a lot for NLP (natural language processing). AutoEncoders are unsupervised, they learn patterns in the data. $\endgroup$
    – MiniQuark
    Sep 14, 2016 at 8:40
  • 1
    $\begingroup$ Alright then :) $\endgroup$
    – Firebug
    May 4, 2019 at 17:06
26
+50
$\begingroup$

Within the fields of adaptive signal processing / machine learning, deep learning (DL) is a particular methodology in which we can train machines complex representations.

Generally, they will have a formulation that can map your input $\mathbf{x}$, all the way to the target objective, $\mathbf{y}$, via a series of hierarchically stacked (this is where the 'deep' comes from) operations. Those operations are typically linear operations/projections ($W_i$), followed by a non-linearities ($f_i$), like so:

$$ \mathbf{y} = f_N(...f_2(f_1(\mathbf{x}^T\mathbf{W}_1)\mathbf{W}_2)...\mathbf{W}_N) $$

Now within DL, there are many different architectures: One such architecture is known as a convolutional neural net (CNN). Another architecture is known as a multi-layer perceptron, (MLP), etc. Different architectures lend themselves to solving different types of problems.

An MLP is perhaps one of the most traditional types of DL architectures one may find, and that's when every element of a previous layer, is connected to every element of the next layer. It looks like this:

enter image description here

In MLPs, the matricies $\mathbf{W}_i$ encode the transformation from one layer to another. (Via a matrix multiply). For example, if you have 10 neurons in one layer connected to 20 neurons of the next, then you will have a matrix $\mathbf{W} \in R^{10 \text{x} 20}$, that will map an input $\mathbf{v} \in R^{10 \text{x} 1}$ to an output $\mathbf{u} \in R^{1 \text{x} 20}$, via: $\mathbf{u} = \mathbf{v}^T \mathbf{W}$. Every column in $\mathbf{W}$, encodes all the edges going from all the elements of a layer, to one of the elements of the next layer.

MLPs fell out of favor then, in part because they were hard to train. While there are many reasons for that hardship, one of them was also because their dense connections didnt allow them to scale easily for various computer vision problems. In other words, they did not have translation-equivariance baked in. This meant that if there was a signal in one part of the image that they needed to be sensitive to, they would need to re-learn how to be sensitive to it if that signal moved around. This wasted the capacity of the net, and so training became hard.

This is where CNNs came in! Here is what one looks like:

enter image description here

CNNs solved the signal-translation problem, because they would convolve each input signal with a detector, (kernel), and thus be sensitive to the same feature, but this time everywhere. In that case, our equation still looks the same, but the weight matricies $\mathbf{W_i}$ are actually convolutional toeplitz matricies. The math is the same though.

It is common to see "CNNs" refer to nets where we have convolutional layers throughout the net, and MLPs at the very end, so that is one caveat to be aware of.

$\endgroup$
4
  • 1
    $\begingroup$ +1 Great answer, I just learned about toeplitz matricies. ;-) $\endgroup$
    – MiniQuark
    Sep 16, 2016 at 14:50
  • $\begingroup$ @MiniQuark Thanks! Yes - you can write out a convolution as a multiplication by a toeplitz matrix. :) $\endgroup$ Sep 16, 2016 at 18:29
  • $\begingroup$ @TarinZiyaee and MiniQurak Can you guys please suggest me any good book/journal article/tutorial for the beginner to learn about the Deep learning in detail. $\endgroup$ Sep 20, 2016 at 2:18
  • $\begingroup$ CNNs are not necessarily deep (-1) $\endgroup$
    – Firebug
    Sep 27, 2018 at 13:07
7
$\begingroup$

Deep learning = deep artificial neural networks + other kind of deep models.

Deep artificial neural networks = artificial neural networks with more than 1 layer. (see minimum number of layers in a deep neural network or Wikipedia for more debate…)

Convolution Neural Network = A type of artificial neural networks

$\endgroup$
5
  • $\begingroup$ Fair enough, Deep Learning also includes "Multilayer kernel machines", and CNNs can be shallow. :) $\endgroup$
    – MiniQuark
    Sep 14, 2016 at 19:01
  • $\begingroup$ @MiniQuark yep. $\endgroup$ Sep 14, 2016 at 19:03
  • $\begingroup$ artificial neural networks with more than 1 layer..... I'm not sure a network with just 2 layers is called a deep network. $\endgroup$
    – SmallChess
    Sep 16, 2016 at 5:29
  • 1
    $\begingroup$ @StudentT Maybe, I don't work in marketing ;) en.wikipedia.org/w/… $\endgroup$ Sep 16, 2016 at 15:08
  • 1
    $\begingroup$ This should be the top answer, because CNNs are not necessarily deep, and Deep learning is not only about ANNs (in the ordinary sense). $\endgroup$
    – Firebug
    Sep 27, 2018 at 13:08
6
$\begingroup$

This slide by Yann LeCun makes the point that only models with a feature hierarchy (lower-level features are learned at one layer of a model, and then those features are combined at the next level) are deep.

A CNN can be deep or shallow; which is the case depends on whether it follows this "feature hierarchy" construction because certain neural networks, including 2-layer models, are not deep.

Yann LeCun's definition

$\endgroup$
0
3
$\begingroup$

Deep learning is a general term for dealing with a complicated neural network with multiple layers. There is no standard definition of what exactly is deep. Usually, you can think a deep network is something that is too big for your laptop and PC to train. The data set would be so huge that you can't fit it into your memory. You might need GPU to speed up your training.

Deep is more like a marketing term to make something sounds more professional than otherwise.

CNN is a type of deep neural network, and there are many other types. CNNs are popular because they have very useful applications to image recognition.

$\endgroup$
3
  • $\begingroup$ I would like to kindly object to a few of these statements: a network is generally considered deep when it has more than one hidden layer, and most people would agree that having more than 10 hidden layers is definitely deep. It is quite possible to train deep networks on your laptop, I do it all the time. The dataset does not have to be huge, in particular if you use transfer learning (ie. reuse layers from a pretrained network), and/or data augmentation. There is certainly a lot of hype around the word "deep", but there are dedicated techniques for deep learning so it's not just hype. :) $\endgroup$
    – MiniQuark
    Sep 16, 2016 at 14:58
  • $\begingroup$ @MiniQuark With data augmentation, your data set will still be large, just that you start off with something small... $\endgroup$
    – SmallChess
    Sep 16, 2016 at 15:01
  • $\begingroup$ Good point, you're right. What I meant to say is that the dataset would fit in memory since you would generate most of the data on the fly. $\endgroup$
    – MiniQuark
    Sep 16, 2016 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.