# What is exact 'learning' on a deep learning model?

This is something I do not understand.

Consider a regular Machine Learning model. I have a lot of pictures of cats and dogs and I feed them to the model and train.

I am new to machine learning but I think that in this case, during the training, the system will try to find some mathematical relations between the pixels of all images that will make possible to identify cats and dogs.

Now let's talk about deep learning.

I create a model that is basically a series of mathematical "formulas" that can detect something.

Imagine I was not using machine learning but rather, just plain programming. I can create a complex program that will take the input parameters using some kind of weight and determine if the image is of a cat or dog.

I can use the same formulas in a model to create a deep learning model.

If a deep learning model is based on "formulas" that I add, how exactly is this different of having plain programming to do the same task? They say the model will train based on the relations, but what will the model learn exactly? Isn't based on a fixed set of rules?

I don't see how better will a deep learning model be from regular programming, starting on the principle that I create the same rules for both.

I understand that the deep learning "formulas" will not change, I mean the weights, right?

• The weights are a function of the data. RTM. – generic_user Feb 3 '19 at 10:06
• ???????????????? – Duck Feb 3 '19 at 10:08

The question is a bit unclear, but here is an attempt at an answer based on my understanding of the question:

First thing that might hinder some understanding is the (in my opinion) false distinction between "regular machine learning" and "deep learning". In terms of what is going on, fundamentally, there is little difference.

So instead let's think about the simple model that learns to distinguish cats and dogs. You take all the dogs, and get an average picture of a dog. Then you take all the cats and get the average picture of a cat. Then, after a new picture of unknown class comes in - you compare it to the average cat and an average dog. And assign to the class that is more similar.

This already is a classification algorithm called "nearest centroid". And it's already "learning" . Based on these averaged cat and dog - you can then compute the separating hyperplane going through the middle between them. That's where you get the "weights" for each pixel.

You could specify those weights by hand, sure. For example after doing the calculations on paper, or just guessing based on eye estimates. But it is a lot of work. And it gets harder with more complex models that takes the relations between pixels into account (like linear discriminant analysis) or introduces a lot of non-linearities (like deep neural network) or that is based on separate instances instead of average cat and dog (like k-nearest neighbour classifier).

In summary: the weights, once estimated, are fixed. This is the optimisation problem - finding a minimum/maximum. Once the solution is obtained - it is fixed. And if you write the same weights down by hand - there will be no difference.

: A quick note about terminology: statistics typically refers to "learning" as "estimating", which I think is more clear. As the word "estimate" includes a possibility that the "trained" model might be false. "learning" suggest that the model got closer to some truth, which might not be the case.

• ok, thanks but when I write a deep learning model I have to right these weights, right? and in the other case the weights are found by the algorithm. – Duck Feb 3 '19 at 12:18
• Are you talking about the initial weights that you have to pass in? If so - these are typically required by incremental optimisation procedures like gradient descend, where you cannot find the global minimum/maximum. Instead you try multiple initiations of solutions and try to iteratively improve them. So you will supply the initial weights, but they will change during training of the network. – Karolis Koncevičius Feb 3 '19 at 12:22
• Alternatively you might be talking about the so called "hyper-paramters". These are the things that change your model in some way. So for example the learning step or the number of iterations or the number of layers and number of nodes in each layer. These you have to pass in, but they are not the solution. Instead choices like this restrict the types of solutions you will be "looking over". Think about it like this: if you specify a model y=ax - then that will optimise "a" as a line. If you had y=ax + bx^2 - that would search for parabolas. Finding a and b is "learning" in this case. – Karolis Koncevičius Feb 3 '19 at 12:27