I have problems with understanding the sub-areas of AI and how it works.

AI has the sub-area Machine Learning (ML), in which learning algorithms are used. Supervised/unsupervised learning takes place in this area. Learning algorithms are, for example: various regressions, SVM, neural network.

ML also has a sub-area - Deep Learning (DL). Here the learning happens through Artificial neural networks (ANN) with hidden layers.

BUT I thought neural networks are learning algorithms and we have already used them in the ML sub-area.

  1. Is the neural network in ML different from that in DL?
  2. Does DL always work with the neural network?
  3. If you work with SVM instead of neural network in ML, do you still work with neuronal network in DL?
  4. Or does DL ALWAYS work with neural network?
  • 3
    $\begingroup$ Deep learning is not strictly about ANN, it can be any model which is "deep" - hidden layers. ANN are in ML, ANN which have hidden layers are in deep learning. $\endgroup$ – user2974951 Apr 6 at 8:53
  • $\begingroup$ why does a dedicated machine learning package like sci-kit learn omit ANN and deep learning algorithms? to use them, people are forced to turn to keras and tensorflow which don't have the same overlap with sklearn $\endgroup$ – develarist Sep 9 at 20:22

This is just terminology, no need to think about it too much as different people classify different areas into different categories. For example a lot of statisticians would consider machine learning to be a sub-area of statistics, people from AI would consider machine learning to be a sub area of AI research, and people working with computer science consider it to be a sub-area of computer science.

With this in mind, the thing to understand is that "Deep Learning" is not a distinct area from "Machine Learning", but a part of it. In the same way that "Building Bridges" is a sub-part of "Mechanics" which is a sub-part of "Physics". With the context of your question - neural networks vs deep neural networks, it is a bit like asking how long bridges are distinct from shorter bridges. Different tools and techniques are involved but the concept is the same.

So, your questions:

1) Neural networks are not different, they just typically have to have more parameters (be "bigger") for them to be labelled "deep" neutral networks.

2) Not necessarily, neural networks themselves, loosely, can be thought of as multiple logistic regressions stacked on top of each other. Any time you create a model and feed its results to another model and then another model, etc, and try to "train" those models together, you can consider such architecture to be "deep".

3) Typically if you will use the term "deep learning" everyone will assume you are talking about neural networks, because that is the current trend and because the term "deep learning" was first applied to neural networks. So if you use any other architecture you will have to specify it to not confuse others.

4) Answered by 2)

| cite | improve this answer | |
  • $\begingroup$ Thank you for your answer. If I use the SVM learning algorithm in an AI to divide data into categories, what do I have to consider about deep learning? I have to prepare a presentation for the project week and want to understand the relationship between SVM and deep learning $\endgroup$ – review Apr 6 at 9:54
  • 1
    $\begingroup$ @review depends on the presentation. If it is about SVM only then do not mention deep learning at all. If you are going to mention something - first thing to mention would be ensemble models where SVM can be a part of ensemble. If you want to drag SVM into "deep learning" territory - mention that it can be integrated into a neural network (i.e. as described by this paper) But at that point your presentation is no longer about SVM. $\endgroup$ – Karolis Koncevičius Apr 6 at 10:02

Agreed with Karolis' answer in "there are no hard boundaries". In addition,

  1. It's the same architecture of course. However, although we don't have a hard threshold on the number of layers for a neural network to be deep, in DL, we're more interested neural networks with large number of layers, instead of 1 or 2.

  2. Typically, yes. See the wikipedia page for example:

Deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning.

But, this doesn't mean it'll be always. A lot of new architectures have been developing lately, e.g. Graph Neural Networks that learns over arbitrary graph structures. This is an extension of neural-nets to graphs, but also significantly differ from the fully connected neural nets we're accustomed to. But, not all of these new architectures have to fit in under ANN topic and we might need to extend the definition in the near future.

  1. Not sure what you've asked.
| cite | improve this answer | |

I would second user2974951's comment (+1). Deep Learning entails stacking or layering within a methodology.

Points 1, 2 & 4 have been fully answered by Karolis (+1).

Regarding point 3: There are works that combined SVM and DNN. (e.g. Tang (2013) Deep Learning using Linear Support Vector Machines where it shows very promising results in replacing with L2-SVM an softmax activation function, or Jiu (2017) Nonlinear Deep Kernel Learning for Image Annotation where multiple kernel learning is presented within a deep learning framework.) In addition to that kernel methods like Gaussian Processes have also seen a resurgent when effectively stacked on top of each other (e.g. see Damianou & Lawrence (2013) Deep Gaussian Processes for fully recurrent way of stacking GPs or Dunlop et al. (2018) How Deep Are Deep Gaussian Processes? for a more in-depth discussion).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.