Given a training set of $(X, Y )$'s where the $X$'s are the source variables and the $Y$'s are the targets, derive an estimator that minimizes the mean squared error between target values and corresponding predictions by the estimator.
Solution
Denote our estimator for particular $x$ as $\theta(x)$ and take $L(y,\theta, x) = |y - \theta(x)|^2$. The total loss will be defined as \begin{align} E_{XY}[L(Y,\theta, X)] &= \int_{\mathbb{R}}\int_{\mathbb{R}}L(y,\theta, x)p(x,y)dxdy\tag{1}\\ &= \int_{\mathbb{R}}\int_{\mathbb{R}}L(y,\theta, x)p(y|x)p(x)dxdy\tag{2}\\ &= \int_{\mathbb{R}}\Big[\int_{\mathbb{R}}(\theta(x) - y)^2p(y|x)dy\Big]p(x)dx\tag{3}\\ \end{align} Now: \begin{align} \frac{dE_{XY}[L(y,\theta, x)]}{d\theta(x)} &=\int_{\mathbb{R}}\frac{d}{d \theta (x)}(\theta(x) - y)^2p(y|x)dy\tag{4}\\ &= 2\int_{\mathbb{R}}(\theta(x) - y)p(y|x)dy\tag{5}\\ &= 2\theta(x)\underbrace{\int_{\mathbb{R}}p(y|x)dy}_{ = \int_{\mathbb{R}}p(y)dy = 1} - 2\int_{\mathbb{R}}yp(y|x)dy\tag{6}\\ &= 2\theta(x) - 2\int_{\mathbb{R}}yp(y|x)dy\tag{7}\\ \end{align} Setting $\frac{dE_{XY}[L(y,\theta, x)]}{d\theta(x)}$ to 0 yields $$ \theta(x) = \int_{\mathbb{R}}yp(y|x)dy\tag{8} $$ My questions
- How do we justify go from (3) to (4) (step by step)?
- In (6) how can we justify that $\int_{\mathbb{R}}p(y|x)dy = \int_{\mathbb{R}}p(y)dy = 1$?