28
$\begingroup$

I received preliminary materials on deep learning in my class. It was written as follows. This raised me the question of the basic meaning of the word deep learning.

Deep learning is a machine learning method using a multi-layer neural network.

  1. Is a neural network essential for deep learning?
  2. Isn't it possible to do deep learning without a neural network by using PCA? (Example: PCANet)

I'm confused by similar terms like deep learning and deep neural networks. I have a month to start school and I can't contact my teacher during that time. I would appreciate it if you could tell me.

Improve this question
$\endgroup$
1
  • 5
    $\begingroup$ I think this question is almost impossible to answer, due to this one. Anything that's a deep hierarchical computational graph that is modeling something will be called a deep neural network, by one account or another. It does not even need to be differentiable to get that treatment. $\endgroup$
    – Firebug
    Commented Apr 4, 2021 at 15:04

6 Answers 6

Reset to default
31
$\begingroup$

This is a good question.

Is a neural network essential for deep learning?

Yes, your teacher provided you with a correct definition of deep learning. You can still do machine learning (a broader category) without neural networks, but you need a neural network for it to qualify as 'deep learning'.

Isn't it possible to do deep learning without a neural network by using PCA? (Example: PCANet)

Based on the answer to the last part, no. It by definition wouldn't be 'deep' anymore. PCANet is actually a neural network, by the way.

PCA, on the other hand, isn't 'deep'. If you stack several layers of PCA on top of each other, then there is an equivalent single-layer PCA you could have done, because composing those linear transforms will just give you another linear transform.

Improve this answer
$\endgroup$
8
  • 10
    $\begingroup$ There is a technique called 'deep forests' that stacks random-forest classifiers, and which would have a reasonable claim to be 'deep learning': arxiv.org/abs/1702.08835 $\endgroup$
    – Thomas Lumley
    Commented Apr 3, 2021 at 5:54
  • 2
    $\begingroup$ But what for @ThomasLumley? To bring more hype to using random forest? $\endgroup$
    – Tim
    Commented Apr 3, 2021 at 15:41
  • 5
    $\begingroup$ @Tim One could just as well ask "what for" about "deep neural networks". What makes "deep" neural networks substantively different from non-deep ones to the extent it justifies a special label? Is it just hype, or is there something else? -- My understanding is that "deep forests" are predicated on the opinion that there is something to "deep", and are an attempt to see if those same qualities can also be implemented with with a decision tree based methods. Whether they succeed or not :shrug:. $\endgroup$
    – R.M.
    Commented Apr 3, 2021 at 17:54
  • 3
    $\begingroup$ "What makes 'deep' networks substantively different from non-deep ones?" For starters, the universal approximation theorem. More info in these lecture slides. $\endgroup$
    – Arya McCarthy
    Commented Apr 3, 2021 at 18:02
  • 9
    $\begingroup$ @AryaMcCarthy Universal approximation theorems apply to both "shallow" (one hidden layer) and deep neural networks, so the UAT cannot explain why the two should be substantively different. One important difference (which is mentioned in your link) is that the number of nodes required to approximate certain functions is exponentially larger for shallow networks compared to deep networks. $\endgroup$
    – Eric Perkerson
    Commented Apr 5, 2021 at 0:32
24
$\begingroup$

I'm going to disagree with the other answers. Fundamentally, I would say that deep learning is defined by a hierarchy of learned representations, and not by which particular model is used to define these representations. Indeed, this is how Goodfellow et al define it in the introductory section of their text Deep Learning (neural networks are not mentioned until later).

In other words, the key with deep learning is that we are effectively learning a series of transformations of our data. Typically, these transformations define a neural network, with the activations of each layer serving as the transformed input data. However, this need not be the case. Deep Gaussian processes, for instance, have been gaining some attention in the research community.

However, if I were teaching an introductory class, I would feel perfectly comfortable using your professor's definition: in practice, people overwhelmingly use neural networks for deep learning.

You can do deep learning with basically any nonlinear model. PCA, being linear, does not qualify (though nonlinear analogues of PCA, such as kernel PCA, can qualify, see this article).

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am a bit torn with this is answer because it is a very good point but probably easy to misinterpret. Even in the example given the whole idea is that we have "layers" and for all intended purposes, "Deep GPs" could be called "Stacked GPs". But we would not really call "stacking" (in the context of ML) an application of Deep Learning because it would be misleading. The concept of "Depth" is pretty loosely defined. The use of the term "Deep" carries some buzz around it. I don't think this post disagrees with the other answers but rather that it critically complements them... You get my +1. $\endgroup$
    – usεr11852
    Commented Apr 3, 2021 at 22:38
  • 1
    $\begingroup$ Agreed, multi-layer Bayesian networks could also qualify as "deep". $\endgroup$
    – madprogramer
    Commented Apr 4, 2021 at 15:07
  • $\begingroup$ Stacked normalizing flows could be considered deep as well. So you do not necessarily need a neural network, though it is the most convinient and commonly used. $\endgroup$
    – Marcel Braasch
    Commented Jun 1, 2021 at 11:40
6
$\begingroup$

This answer depends on the definition of artificial neural network (ANN) you take to be true. See my question here: What *is* an Artificial Neural Network?. Therefore, no objective answer can be given since:

  1. To accommodate all definitions of ANNs, they are simply defined as arbitrary computational graphs, with tunable parameters (even if they are not tuned, random neural networks are still ANNs)
  2. Differentiability is not required (for example, genetic algorithms and other metaheuristics have been used to learn ANNs)

I will however, propose a counterfactual approach here that, in my experience, at least makes some people to conclude that deep learning is not solely based on ANNs:

  1. Do you consider hierarchical hidden markov models (HHMM)/hierarchical Bayesian networks deep learning?
  2. Do you consider them to be neural networks?

If you answered 'yes' and 'no', then the overall answer is definitely 'no', for you there is a deep learning model that is not a neural network. Other combinations of answer will lead to undefined overall answers.

Improve this answer
$\endgroup$
5
$\begingroup$

Deep learning is machine learning done using “deep” neural networks, i.e. such that have multiple (>2) layers. So you cannot do it without neural networks. For using other kinds of machine learning just use “machine learning” term, that includes neural networks as well.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ By definition, then, deep learning requires neural networks. However, a neural network is a fairly straightforward thing to implement in software and there's nothing magic about it. It would also be straightforward to implement something different in software that performed a similar job, and we could have a lively debate as to whether it was a neural network or not :-) In other words, deep learning only requires neural networks because that's the definition of the term, not because that behaviour can only be achieved using neural networks. $\endgroup$
    – Frog
    Commented Apr 4, 2021 at 1:46
  • $\begingroup$ @Frog No reason to re-use the “deep learning” name for something else if there’s already a term for it. By same logic, NNs are general approximators, so a non-deep NN would be considered as “deep” if it approximated the deep one? $\endgroup$
    – Tim
    Commented Apr 4, 2021 at 9:41
1
$\begingroup$

To answer from a different perspective, one philosophical point may help: the concept of learning in deep learning.

In deep learning, there are multiple learning steps. In the first step, the data input is 'converted' (or learned) into a synthetic intermediate output (a bit higher abstraction, loosely speaking). Then in each step, the previous output is progressively learned (or 'transformed') into higher abstraction features, which may or may not be comprehensible to humans. Loosely speaking, the combination of these layers will approximate the 'model' for you, we don't need to specify any model or hypothesis beforehand. This is the same 'learning' concept as in cognitive science: find/construct higher abstraction from raw input.

So, do we need NN for deep learning? Yes, in practice. With my limited knowledge, I would say this is the most convenient and efficient way to do it. In theory, it depends. If you build a 'learning' framework in which your model can create 'deep' abstraction (iteratively increasing) from raw input to achieve the task at hand, it may count as deep learning.

In practice, it's a bit simplistic to view Deep Learning as just a vanilla multi-layer neural network. More than often, the workflow includes different NN-components and other transformation layers, each part can be wildly different from the next.

I hope this answer provide some useful insights without the need to use technical terms.

Improve this answer
$\endgroup$
0
$\begingroup$

Deep learning is a subset of machine learning, which is a field dedicated to the study and development of machines that can learn, and the goal is of deep learning is to achieve eventually attain general artificial intelligence. Neural network is just one of the biological inspired model. In simple terms I would define deep learning a subset method in machine learning primarily using artificial neural networks which are inspired by human brain. See the following diagram.Artificial Intelligence Figure

Classical machine learning uses classical statistics/mathematical modelling while deep learning use biological models aka neural networks.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.