The expected regression loss is given as:$$E[L]=\int\int \{y(\mathbf x)-t\}^2 p(\mathbf x,t)d\mathbf xdt$$ To minimise the expected loss,Euler Lagrange equation is used which goes like this in the general form:$$ \frac{\delta F}{\delta y}=\frac{d}{dx}\left( \frac{\delta F}{\delta y'}\right)$$ where $F$ is the functional i.e function of function $\mathit y(x)$. In our case $F$ would correspond to $E$ and $\mathit y$ to $L$. Please show a step wise application of this equation to yield the following expression $$\frac{\delta E[L]}{\delta \mathit y(\mathbf x)}=2 \int\{\mathit y(\mathbf x)-t\}p(\mathbf x,t) dt=0$$ Also,please suggest a good reference for all the calculus used in Machine Learning.


I will carry out the steps in a way I hope is clear enough to indicate what assumptions must be made about $y,$ $p,$ and $\delta$ to justify the steps.

When a function $y$ is a local minimum of a functional $\mathcal L,$ adding a sufficiently small multiple $h$ of a "test function" $\delta$ cannot decrease the value of the functional. (Usually a test function is assumed to be arbitrarily smooth and of compact support; in more complex situations it may have to satisfy constraints imposed by boundary conditions.) That is, writing

$$\mathcal{L}[y] = \iint \left(y(x)-t\right)^2\,p(x,t)\,\mathrm{d}x\mathrm{d}t,$$

it must be the case that for all $|h|$ in some small neighborhood $U$ of $0$ (which may depend on $\delta$),

$$\eqalign{ 0 &\le \mathcal{L}[y+h\delta] - \mathcal{L}[y] \\ &= 2h\iint \delta(x) \left(y(x)-t\right)\,p(x,t)\,\mathrm{d}x\mathrm{d}t + h^2 \iint \delta^2(x) \,p(x,t)\,\mathrm{d}x\mathrm{d}t .}\tag{1}$$

Given $\delta,$ the right hand side is a quadratic function of $h$ with a zero at $h=0.$ (This assertion follows from the fact that the integral of $\delta^2$ is finite, which guarantees both integrals in $(1)$ exist and are finite provided $y$ and $p$ are not too "badly behaved," which is implicitly assumed in the question.)

In order to be non-negative throughout $U$, this zero must be the vertex of the quadratic; that is,

$$0 = 2\iint \delta(x) \left(y(x)-t\right)\,p(x,t)\,\mathrm{d}x\mathrm{d}t$$

This says the function

$$x \to \int \left(y(x)-t\right)p(x,t)\,\mathrm{d}t$$

is orthogonal to all test functions $\delta,$ implying (by virtue of some assumed "not bad behavior" of $p$) it is almost everywhere zero, QED.

Generally, to find the derivative of a functional $\mathcal L,$ you proceed as in ordinary differential Calculus to form the difference quotient as in $(1)$

$$\frac{\mathcal{L}[y+h\delta] - \mathcal{L}[y]}{h} = \iint \delta(x)\, 2\left(y(x)-t\right)\,p(x,t)\,\mathrm{d}x\mathrm{d}t + o(h).$$

The right hand side applies the linear operator

$$D[\delta] = \int \delta(x)\,\left[2\int (y(x)-t)\,p(x,t)\,\mathrm{d}t\right]\,\mathrm{d}x$$

represented by the generalized function

$$x \to 2\int (y(x)-t)\,p(x,t)\,\mathrm{d}t$$

to $\delta$ and adds an error term $o(h)$ that is vanishingly small compared to $h.$ This conforms with the (usual) definition of the derivative, allowing us to write

$$\frac{\partial \mathcal{L}}{\partial y} = 2\int (y(x)-t)\,p(x,t)\,\mathrm{d}t.$$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.