What is exact 'learning' on a deep learning model?

Question

This is something I do not understand.

Consider a regular Machine Learning model. I have a lot of pictures of cats and dogs and I feed them to the model and train.

I am new to machine learning but I think that in this case, during the training, the system will try to find some mathematical relations between the pixels of all images that will make possible to identify cats and dogs.

Now let's talk about deep learning.

I create a model that is basically a series of mathematical "formulas" that can detect something.

Imagine I was not using machine learning but rather, just plain programming. I can create a complex program that will take the input parameters using some kind of weight and determine if the image is of a cat or dog.

I can use the same formulas in a model to create a deep learning model.

If a deep learning model is based on "formulas" that I add, how exactly is this different of having plain programming to do the same task? They say the model will train based on the relations, but what will the model learn exactly? Isn't based on a fixed set of rules?

I don't see how better will a deep learning model be from regular programming, starting on the principle that I create the same rules for both.

I understand that the deep learning "formulas" will not change, I mean the weights, right?

Answer 1

The question is a bit unclear, but here is an attempt at an answer based on my understanding of the question:

First thing that might hinder some understanding is the (in my opinion) false distinction between "regular machine learning" and "deep learning". In terms of what is going on, fundamentally, there is little difference.

So instead let's think about the simple model that learns to distinguish cats and dogs. You take all the dogs, and get an average picture of a dog. Then you take all the cats and get the average picture of a cat. Then, after a new picture of unknown class comes in - you compare it to the average cat and an average dog. And assign to the class that is more similar.

This already is a classification algorithm called "nearest centroid". And it's already "learning" [1]. Based on these averaged cat and dog - you can then compute the separating hyperplane going through the middle between them. That's where you get the "weights" for each pixel.

You could specify those weights by hand, sure. For example after doing the calculations on paper, or just guessing based on eye estimates. But it is a lot of work. And it gets harder with more complex models that takes the relations between pixels into account (like linear discriminant analysis) or introduces a lot of non-linearities (like deep neural network) or that is based on separate instances instead of average cat and dog (like k-nearest neighbour classifier).

---

In summary: the weights, once estimated, are fixed. This is the optimisation problem - finding a minimum/maximum. Once the solution is obtained - it is fixed. And if you write the same weights down by hand - there will be no difference.

---

[1]: A quick note about terminology: statistics typically refers to "learning" as "estimating", which I think is more clear. As the word "estimate" includes a possibility that the "trained" model might be false. "learning" suggest that the model got closer to some truth, which might not be the case.

Answer 2

It may be easier to dissect your question. You are asking 2 things at once.

First, let's make clear Learning (with machine) vs. Programing:

Say we have a task: assign a class (cat vs. not-cat) to pictures that have no label. We need to find a relationship between the arrangement of certain pixels to the concept 'cat'. This relationship can be a combination of 'formulas' as you said, with some parameters. Here's the kicker: if you hard program it, you need to know the formulas, and the parameters. In contrast, learning (or estimating) means we extract from the data the parameters (supervised learning) AND also the formulas if possible (deep learning). Deep learning is very flexible because it doesn't require a hypothesis (an assumed formula).

Now, Deep learning vs. 'Shallow' learning (regression, for example):

In a regression, there is only 1 step learning (estimation). The input is the data, the output is the 'relationship' or formula's set of parameters. And also, you need to know (or assume) the formula beforehand.

In deep learning, there are multiple layers, each layer is a learning step. In the first step, the data input is 'converted' (or learned) into a synthetic intermediate output (a bit higher abstraction, loosely speaking, you look for combination of pixels instead of each single pixel). Then in each step, the input is progressively learned ( or 'transformed') into higher abstraction features. These features (which may not be comprehensible to humans) will be used in the last step of 'learning' to produce the output, which is the probability in our example. Loosely speaking, the reason we have multiple layers is that the combination of these layers will approximate the 'formula' for you, we don't need to specify any hypothesis beforehand. This is the meaning of 'learning', in ML or cognitive science: find higher abstraction patterns.

In short, deep learning is a process with multiple learning steps, the last step can be a simple regression or any model. The intermediate steps just transform data input into 'higher abstraction' input. Another example of deep learning is NLP, where words are transformed progressively into vectors of numbers with increasing abstraction, such that in the end, the vectors of number can actually represent syntactic (grammar) and semantic (logic) meaning.