# 'Learning' Part in Machine Learning?

I am trying to understand better the 'Learning' part in "Machine Learning', and pinpoint when it happens. Will use ** to highlight the key part.

This is what I think it is: It consists of tweaking the parameters of an algorithm. Let me apply it to regression specifically , since I understand it (more or less).

1) We are given data points $(x_1, y_1),.....,(x_n, y_n)$ which we separate into test data and and training data. We have a loss function f and a choice of threshold C . We want to satisfy the condition : || Residual ||< c , where Residual is the difference between the model and actual data ( i.e., Predicted - Observed).

2) If || Residual ||< c is satisfied, we exit.

3) If ||Residual ||$\geq c$ ** . Then the regression model ( meaning here the choice of parameters ) is rejected and the computer ** Learns a better choice of parameters by gradient descent.

4) New parameters are used an we test again for whether || Residual ||

5) We go to 3)

Is this correct? If so, is the idea the same for different algorithms? If not, what I am missing? Thanks. the choice of parameters or coefficients

2) We use the test data to run our

The "learning" part of machine learning is . You've described a particular optimization problem.

• Thanks. Did I identify correctly the 'learning' part, though? Meaning, does the Optimization refer to finding parameters that minimize the residuals to the extent we want (meaning below our choice of threshold)? Still, not 100% sure in what sense the machine is learning. Could you please expand? – gary Jun 18 '18 at 2:40
• The machine is "learning" exactly in the sense that you described: by optimization, the procedure to minimize a loss. Minimizing them "to the extent that we want" is called a "termination condition" in optimization theory. – Sycorax says Reinstate Monica Jun 18 '18 at 2:50
• How does this change with classification techniques or with methods like neural networks? Is the tweaking technique also GD? – gary Jun 20 '18 at 1:40
• The "tweaking technique" is otherwise known as a "parameter update," and it's generic to numerical optimization. Gradient descent is an example of numerical optimization, and GD can be used to estimate neural network parameters, among other methods. Really, there's no substitute for actually reading a book on the subject. Numerical Optimization by Nocedal and Wright is a good one. So is Convex Optimization by Boyd and Vandenbergh. Or Elements of Statistical Learning if you're primarily concerned with machine learning methods. – Sycorax says Reinstate Monica Jun 20 '18 at 3:42
• I agree, but I like to do both bottom-up and top-down reading. Thanks for reply. – gary Jun 20 '18 at 14:18

The idea is this. You have a set of data input pairs (x, y), or tuples, as I will call them (in the simplest case). You want to (for example) predict y - or something about y - based on knowing the x it goes with. You split it into a learning group and a test group.

Now you could be trying to 'teach' any number of approaches or algos. It could be a neural network; it could be an SVM; it could also be old-fashioned regression.

However the mechanism works, you feed it x's and tell it if it's prediction is wrong (or how wrong it is). The goal is for the algo to find a 'best' way to predict the y from any x. Then you test in on the test data to make sure it really does predict (the big risk being overfitting of the training data). #

Broadly speaking, there are two kinds of algos. Ones like regression don't really, in my mind, involve 'machine learning'. It's just applying standardized math to solve a known optimization problem (least squares).

But if you have a neural net it is a different beast, because the net does change and evolve to move toward it's best possible algo given the net design.

• It seems like the distinction you draw between regression and neural networks, whether the latter is machine learning while the former is not, is an example of the sorites paradox. Specifically, a neural network without a hidden layer and using linear activations is identical to an ordinary regression problem. Is there any way you could make that distinction more rigorous? – Sycorax says Reinstate Monica Jun 18 '18 at 3:30
• It's a fine point mathematically. OLS is the solution to a specific problem with specific assumptions, namely, the linear relationship between two (normally-distributed) random variables. As such, the solution, mathematically, is know a priori and is guaranteed and certain (absent a defect in the data). A one layer NN will get the same answer, and it can be shown it is the same answer and an global (not local) optimization. A hidden-layer may converge to a local but not global optimum. An SVM will give a global solution, though the kernel choice matters (to avoid over-fitting again). – eSurfsnake Jun 18 '18 at 4:45
• Of course, even the neural net with no hidden layers only agrees with OLS because of (1) certain assumptions about the form of the loss function and (2) the known mechanics of the back-propagation algo being used. Pick other loss functions, or a different way to get NN weights, and you will get a different answer. For example, imagine the "error" is not the (sum of) square(s) of deviations from the mean, but the absolute value of departure from the mean./ – eSurfsnake Jun 18 '18 at 4:49
• @eSurfsnake: Could you elaborate on how learning comes about on NNs? Do we also use Gradient Descent/Ascent? – gary Jun 18 '18 at 17:31
• Yes. Look, in the end, all of these techniques are optimizations of one sort or another, and we have the optimization machinery pre-built. So, in that sense, the machine never 'learns' anything. The learning really means the act of splitting the data into a training set and a test set, running the machine on the learning set, then using the test set to see if it worked right (e.g., didn't get stuck in a local max or min, or over-fit, etc.). I find the term machine learning a bit overused and sci-fi feeling. – eSurfsnake Jun 19 '18 at 21:14