minimum value of $E[Y - g(X)]^2$

Question

When is $E[Y - g(X)]^2$ at it's minimum value over possible transforms $g$? I guess it's when $g(x) = E[Y|X]$. But i'm unable to prove it. Any ideas or hints how can i start the proof? (If i'm right about the answer)

Answer 1

$$\begin{align} [Y-g(X)]^2 &= [Y-\mathbb{E}[Y \mid X]+\mathbb{E}[Y \mid X]-g(X)]^2 \\ &= [Y-\mathbb{E}[Y \mid X]]^2+[\mathbb{E}[Y \mid X]-g(X)]^2+2[Y-\mathbb{E}[Y \mid X]][\mathbb{E}[Y \mid X]-g(X)] \end{align}$$ Now, by double expectation, $$\mathbb{E}[Y-g(X)]^2 = \mathbb{E}[\mathbb{E}\left\{[Y-g(X)]^2 \mid X\right\}]\text{.}$$ Looking at the third term of $\mathbb{E}[Y - g(X)]^2$ above, notice that $$\mathbb{E}\left\{Y-\mathbb{E}[Y \mid X] \mid X\right\}= \mathbb{E}[Y \mid X] - \mathbb{E}[\mathbb{E}[Y \mid X]\mid X]=\mathbb{E}[Y \mid X] - \mathbb{E}[Y \mid X] = 0$$ so thus, we end up with $$\mathbb{E}[Y - g(X)]^2 = \mathbb{E}[Y-\mathbb{E}[Y \mid X]]^2+\mathbb{E}[\mathbb{E}[Y \mid X]-g(X)]^2\text{.}\tag{1}$$ Note that in $(1)$, the only term dependent on $g(X)$ on the right side is $\mathbb{E}[\mathbb{E}[Y \mid X]-g(X)]^2$. Since this is the expectation of a squared quantity, we know that $$[\mathbb{E}[Y \mid X]-g(X)]^2 \geq 0$$ which means that minimization happens when $$\mathbb{E}[Y \mid X]-g(X) = 0$$ or $$g(X) = \mathbb{E}[Y \mid X]\text{.}$$

Answer 2

Add and substract $E(Y|X)$ in your expression and condition to $X$ to prove the undebraced expression is equal to zero. \begin{align*} & E(Y - g(X))^2 = \\ &E (Y- E(Y|X))^2 + 2\underbrace{E\left[(Y - E(Y|X) (E(Y|X) - g(X))\right]}_{=0} + E(E(Y|X) - g(X))^2 \\ &\geq E(Y- E(Y|X))^2 \end{align*}

To reply to the comments in the question here is why the underbraced expression is equal to zero.

$$E\left[(Y - E(Y|X) (E(Y|X) - g(X))\right] = E \{ E\left[(Y - E(Y|X) (E(Y|X) - g(X))|X\right] \} = E \{(E(Y|X) - g(X)) \underbrace{E \left[(Y - E(Y|X) |X\right]}_{=0} \} = 0 $$ Where the term $E(E(Y|X) - g(X))$ goes out the conditional expectation to $X$ because it is a function of $X$.

Answer 3

Since answers have already been provided, here is a short intro to conditional expectations as $L^2$ projections, i.e. for an alternative path to conditional expectations in which the question asked is taken more or less as definition.

Intuitively, taking the expected value of $Y$ conditioned on $X$ means averaging $Y$ over events which produce the outcomes of $X$. If we think of the $\sigma$-algebra generated by $X$, $\sigma(X)$, as grouping events leading to the same outcomes of $X$, then we see that we want the conditional expectation $\mathbb{E}[Y|X]$ to be a r.v. over $\sigma(x)$, closest in some sense to the original $Y$.

Therefore, if we work in $L^2(\Omega, \mathcal{A}, P)$, with $\mathcal{A}$ a $\sigma$-algebra over $\Omega$, then the conditional expectation $\mathbb{E}[Y|X]$ has a natural definition as the $L^2$ projection of Y onto $L^2$ of $\sigma(X)$ since $L^2(\Omega, \sigma(X), P)$ is a closed subspace of $L^2(\Omega, \mathcal{A}, P)$:

$$\mathbb{E}[Y|X] = \operatorname{argmin}_g \mathbb{E}[(Y-g(X))^2]$$

over all Borel $g$ such that $g(X) \in L^2$ (recall that $Z$ is measurable wrt. $\sigma(X)$ iff there is some Borel $g$ such that $Z=g(X)$)

The case $L^1$ is not so straightforward because we lack the Hilbert space structure, see e.g. "Jacod, Protter: Probability essentials", Chapter 23.

Conditional expectations can be taken over any sub $\sigma$-algebra of $\mathcal{A}$. For a nice introduction starting from conditioning on $\sigma$-algebras and with pictures and intuition, see "Conditional Expectations without tears".