What happens in the sub-areas of AI? (ML, DL)

Question

I have problems with understanding the sub-areas of AI and how it works.

AI has the sub-area Machine Learning (ML), in which learning algorithms are used. Supervised/unsupervised learning takes place in this area. Learning algorithms are, for example: various regressions, SVM, neural network.

ML also has a sub-area - Deep Learning (DL). Here the learning happens through Artificial neural networks (ANN) with hidden layers.

BUT I thought neural networks are learning algorithms and we have already used them in the ML sub-area.

1. Is the neural network in ML different from that in DL?
2. Does DL always work with the neural network?
3. If you work with SVM instead of neural network in ML, do you still work with neuronal network in DL?
4. Or does DL ALWAYS work with neural network?

Answer 1

This is just terminology, no need to think about it too much as different people classify different areas into different categories. For example a lot of statisticians would consider machine learning to be a sub-area of statistics, people from AI would consider machine learning to be a sub area of AI research, and people working with computer science consider it to be a sub-area of computer science.

With this in mind, the thing to understand is that "Deep Learning" is not a distinct area from "Machine Learning", but a part of it. In the same way that "Building Bridges" is a sub-part of "Mechanics" which is a sub-part of "Physics". With the context of your question - neural networks vs deep neural networks, it is a bit like asking how long bridges are distinct from shorter bridges. Different tools and techniques are involved but the concept is the same.

So, your questions:

1) Neural networks are not different, they just typically have to have more parameters (be "bigger") for them to be labelled "deep" neutral networks.

2) Not necessarily, neural networks themselves, loosely, can be thought of as multiple logistic regressions stacked on top of each other. Any time you create a model and feed its results to another model and then another model, etc, and try to "train" those models together, you can consider such architecture to be "deep".

3) Typically if you will use the term "deep learning" everyone will assume you are talking about neural networks, because that is the current trend and because the term "deep learning" was first applied to neural networks. So if you use any other architecture you will have to specify it to not confuse others.

4) Answered by 2)

Answer 2

I would second user2974951's comment (+1). Deep Learning entails stacking or layering within a methodology.

Points 1, 2 & 4 have been fully answered by Karolis (+1).

Regarding point 3: There are works that combined SVM and DNN. (e.g. Tang (2013) Deep Learning using Linear Support Vector Machines where it shows very promising results in replacing with L2-SVM an softmax activation function, or Jiu (2017) Nonlinear Deep Kernel Learning for Image Annotation where multiple kernel learning is presented within a deep learning framework.) In addition to that kernel methods like Gaussian Processes have also seen a resurgent when effectively stacked on top of each other (e.g. see Damianou & Lawrence (2013) Deep Gaussian Processes for fully recurrent way of stacking GPs or Dunlop et al. (2018) How Deep Are Deep Gaussian Processes? for a more in-depth discussion).

Answer 3

Agreed with Karolis' answer in "there are no hard boundaries". In addition,

1. It's the same architecture of course. However, although we don't have a hard threshold on the number of layers for a neural network to be deep, in DL, we're more interested neural networks with large number of layers, instead of 1 or 2.

2. Typically, yes. See the wikipedia page for example:

Deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning.

But, this doesn't mean it'll be always. A lot of new architectures have been developing lately, e.g. Graph Neural Networks that learns over arbitrary graph structures. This is an extension of neural-nets to graphs, but also significantly differ from the fully connected neural nets we're accustomed to. But, not all of these new architectures have to fit in under ANN topic and we might need to extend the definition in the near future.

3. Not sure what you've asked.