# This is some variational calculus used in Bishop's Pattern Recognition and Machine Learning Book (section 1.5.5) on “Loss function for regression”

The expected regression loss is given as:$$E[L]=\int\int \{y(\mathbf x)-t\}^2 p(\mathbf x,t)d\mathbf xdt$$ To minimise the expected loss,Euler Lagrange equation is used which goes like this in the general form:$$\frac{\delta F}{\delta y}=\frac{d}{dx}\left( \frac{\delta F}{\delta y'}\right)$$ where $$F$$ is the functional i.e function of function $$\mathit y(x)$$. In our case $$F$$ would correspond to $$E$$ and $$\mathit y$$ to $$L$$. Please show a step wise application of this equation to yield the following expression $$\frac{\delta E[L]}{\delta \mathit y(\mathbf x)}=2 \int\{\mathit y(\mathbf x)-t\}p(\mathbf x,t) dt=0$$ Also,please suggest a good reference for all the calculus used in Machine Learning.

I will carry out the steps in a way I hope is clear enough to indicate what assumptions must be made about $$y,$$ $$p,$$ and $$\delta$$ to justify the steps.

When a function $$y$$ is a local minimum of a functional $$\mathcal L,$$ adding a sufficiently small multiple $$h$$ of a "test function" $$\delta$$ cannot decrease the value of the functional. (Usually a test function is assumed to be arbitrarily smooth and of compact support; in more complex situations it may have to satisfy constraints imposed by boundary conditions.) That is, writing

$$\mathcal{L}[y] = \iint \left(y(x)-t\right)^2\,p(x,t)\,\mathrm{d}x\mathrm{d}t,$$

it must be the case that for all $$|h|$$ in some small neighborhood $$U$$ of $$0$$ (which may depend on $$\delta$$),

\eqalign{ 0 &\le \mathcal{L}[y+h\delta] - \mathcal{L}[y] \\ &= 2h\iint \delta(x) \left(y(x)-t\right)\,p(x,t)\,\mathrm{d}x\mathrm{d}t + h^2 \iint \delta^2(x) \,p(x,t)\,\mathrm{d}x\mathrm{d}t .}\tag{1}

Given $$\delta,$$ the right hand side is a quadratic function of $$h$$ with a zero at $$h=0.$$ (This assertion follows from the fact that the integral of $$\delta^2$$ is finite, which guarantees both integrals in $$(1)$$ exist and are finite provided $$y$$ and $$p$$ are not too "badly behaved," which is implicitly assumed in the question.)

In order to be non-negative throughout $$U$$, this zero must be the vertex of the quadratic; that is,

$$0 = 2\iint \delta(x) \left(y(x)-t\right)\,p(x,t)\,\mathrm{d}x\mathrm{d}t$$

This says the function

$$x \to \int \left(y(x)-t\right)p(x,t)\,\mathrm{d}t$$

is orthogonal to all test functions $$\delta,$$ implying (by virtue of some assumed "not bad behavior" of $$p$$) it is almost everywhere zero, QED.

Generally, to find the derivative of a functional $$\mathcal L,$$ you proceed as in ordinary differential Calculus to form the difference quotient as in $$(1)$$

$$\frac{\mathcal{L}[y+h\delta] - \mathcal{L}[y]}{h} = \iint \delta(x)\, 2\left(y(x)-t\right)\,p(x,t)\,\mathrm{d}x\mathrm{d}t + o(h).$$

The right hand side applies the linear operator

$$D[\delta] = \int \delta(x)\,\left[2\int (y(x)-t)\,p(x,t)\,\mathrm{d}t\right]\,\mathrm{d}x$$

represented by the generalized function

$$x \to 2\int (y(x)-t)\,p(x,t)\,\mathrm{d}t$$

to $$\delta$$ and adds an error term $$o(h)$$ that is vanishingly small compared to $$h.$$ This conforms with the (usual) definition of the derivative, allowing us to write

$$\frac{\partial \mathcal{L}}{\partial y} = 2\int (y(x)-t)\,p(x,t)\,\mathrm{d}t.$$