What does residual mean in the context of minimizing a function?

Question

equation 1.2 in PRML: pattern recognition and machine learning

denotes the sum of the squares of the errors between the predictions $y(x_n,w)$ and the corresponding target values $t_n$.

$w^*$ denotes a unique solution for the error function above.

page 6 of that book says

For each choice of M, we can then evaluate the residual value of E($w^*$) given by (1.2) for the training data

is residual a math term or machine learning term? what does residual here mean?

I've searched differentiate residual, the result is more confusing.

could someone please give some explanation about residual in this case (equation 1.2)

please provide a solid reference, such as a textbook, for the definition of residual.

Answer 1

wiki Errors_and_residuals

In statistics and optimization, errors and residuals are two closely related and easily confused measures of the deviation of an observed value of an element of a statistical sample from its "theoretical value". The error (or disturbance) of an observed value is the deviation of the observed value from the (unobservable) true value of a quantity of interest (for example, a population mean), and the residual of an observed value is the difference between the observed value and the estimated value of the quantity of interest (for example, a sample mean). The distinction is most important in regression analysis, where the concepts are sometimes called the regression errors and regression residuals and where they lead to the concept of studentized residuals.

Answer 2

A residual is simply the difference between a model's fitted value and the actual value.

In terms of the usual minimization in machine learning, the training of most regression problems is to optimize a set of parameters that minimize the sum of squared residuals on the training set.