Given a training set of $(X, Y )$'s where the $X$'s are the source variables and the $Y$'s are the targets, derive an estimator that minimizes the mean squared error between target values and corresponding predictions by the estimator.


Denote our estimator for particular $x$ as $\theta(x)$ and take $L(y,\theta, x) = |y - \theta(x)|^2$. The total loss will be defined as \begin{align} E_{XY}[L(Y,\theta, X)] &= \int_{\mathbb{R}}\int_{\mathbb{R}}L(y,\theta, x)p(x,y)dxdy\tag{1}\\ &= \int_{\mathbb{R}}\int_{\mathbb{R}}L(y,\theta, x)p(y|x)p(x)dxdy\tag{2}\\ &= \int_{\mathbb{R}}\Big[\int_{\mathbb{R}}(\theta(x) - y)^2p(y|x)dy\Big]p(x)dx\tag{3}\\ \end{align} Now: \begin{align} \frac{dE_{XY}[L(y,\theta, x)]}{d\theta(x)} &=\int_{\mathbb{R}}\frac{d}{d \theta (x)}(\theta(x) - y)^2p(y|x)dy\tag{4}\\ &= 2\int_{\mathbb{R}}(\theta(x) - y)p(y|x)dy\tag{5}\\ &= 2\theta(x)\underbrace{\int_{\mathbb{R}}p(y|x)dy}_{ = \int_{\mathbb{R}}p(y)dy = 1} - 2\int_{\mathbb{R}}yp(y|x)dy\tag{6}\\ &= 2\theta(x) - 2\int_{\mathbb{R}}yp(y|x)dy\tag{7}\\ \end{align} Setting $\frac{dE_{XY}[L(y,\theta, x)]}{d\theta(x)}$ to 0 yields $$ \theta(x) = \int_{\mathbb{R}}yp(y|x)dy\tag{8} $$ My questions

  • How do we justify go from (3) to (4) (step by step)?
  • In (6) how can we justify that $\int_{\mathbb{R}}p(y|x)dy = \int_{\mathbb{R}}p(y)dy = 1$?

The derivation $$\frac{\text{d}\mathbb E_{XY}[L(Y,\theta, X)]}{\text{d}\theta(x)} $$ is meaningless since $$\mathbb E_{XY}[L(Y,\theta, X)]$$ depends on $\theta$ and $X$ is integrated out. (In other words, there is no $x$.) Since $\theta$ is a function, standard derivation does not apply.

The proper argument to the result is that, in order to minimise (3) in $\theta$, one need minimise $$\mathbb E_{Y|X}[L(Y,\theta, X)|X]=\mathbb E_{Y|X}[L(Y,\theta(x), x)|X=x]$$ for (almost) every value $x$ of the random variable $X$, which leads to consider $$\frac{\text{d}\mathbb E_{Y|X}[L(Y,\theta(x), x)|X=x]}{\text{d}\theta(x)} $$ with equations (5)-(8) being correct.

Furthermore, $$\int_{\mathbb{R}}p(y|x)dy = \int_{\mathbb{R}}p(y)dy = 1$$ is correct because both $p(\cdot|x)$ and $p(\cdot)$ are probability densities, but the central integral is irrelevant and hence confusing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.