3
$\begingroup$

When is $E[Y - g(X)]^2$ at it's minimum value over possible transforms $g$? I guess it's when $g(x) = E[Y|X]$. But i'm unable to prove it. Any ideas or hints how can i start the proof? (If i'm right about the answer)

$\endgroup$
3
  • 1
    $\begingroup$ Welcome to CV, Ju Bc. It looks like you are asking a self-study question. I would recommend editing your question (with the small "edit" link in lower left) to (1) include the [self-study] tag, and to (2) possibly incorporate some of the guidance on how to effectively ask a self-study question given in its tag description. $\endgroup$
    – Alexis
    Dec 26, 2017 at 17:36
  • $\begingroup$ The problem (as well as answers) are incorrect, the criterion should be $E[(Y - g(X))^2|X]$. If the conditioning part is missing, one cannot conclude that the cross product term is 0. $\endgroup$
    – Zhanxiong
    Dec 27, 2017 at 19:19
  • $\begingroup$ Your comment is wrong @Zhangxiong. I will add more detail to my existing answer. $\endgroup$
    – Manuel
    Dec 28, 2017 at 16:08

3 Answers 3

4
$\begingroup$

$$\begin{align} [Y-g(X)]^2 &= [Y-\mathbb{E}[Y \mid X]+\mathbb{E}[Y \mid X]-g(X)]^2 \\ &= [Y-\mathbb{E}[Y \mid X]]^2+[\mathbb{E}[Y \mid X]-g(X)]^2+2[Y-\mathbb{E}[Y \mid X]][\mathbb{E}[Y \mid X]-g(X)] \end{align}$$ Now, by double expectation, $$\mathbb{E}[Y-g(X)]^2 = \mathbb{E}[\mathbb{E}\left\{[Y-g(X)]^2 \mid X\right\}]\text{.}$$ Looking at the third term of $\mathbb{E}[Y - g(X)]^2$ above, notice that $$\mathbb{E}\left\{Y-\mathbb{E}[Y \mid X] \mid X\right\}= \mathbb{E}[Y \mid X] - \mathbb{E}[\mathbb{E}[Y \mid X]\mid X]=\mathbb{E}[Y \mid X] - \mathbb{E}[Y \mid X] = 0$$ so thus, we end up with $$\mathbb{E}[Y - g(X)]^2 = \mathbb{E}[Y-\mathbb{E}[Y \mid X]]^2+\mathbb{E}[\mathbb{E}[Y \mid X]-g(X)]^2\text{.}\tag{1}$$ Note that in $(1)$, the only term dependent on $g(X)$ on the right side is $\mathbb{E}[\mathbb{E}[Y \mid X]-g(X)]^2$. Since this is the expectation of a squared quantity, we know that $$[\mathbb{E}[Y \mid X]-g(X)]^2 \geq 0$$ which means that minimization happens when $$\mathbb{E}[Y \mid X]-g(X) = 0$$ or $$g(X) = \mathbb{E}[Y \mid X]\text{.}$$

$\endgroup$
4
$\begingroup$

Add and substract $E(Y|X)$ in your expression and condition to $X$ to prove the undebraced expression is equal to zero. \begin{align*} & E(Y - g(X))^2 = \\ &E (Y- E(Y|X))^2 + 2\underbrace{E\left[(Y - E(Y|X) (E(Y|X) - g(X))\right]}_{=0} + E(E(Y|X) - g(X))^2 \\ &\geq E(Y- E(Y|X))^2 \end{align*}

To reply to the comments in the question here is why the underbraced expression is equal to zero.

$$E\left[(Y - E(Y|X) (E(Y|X) - g(X))\right] = E \{ E\left[(Y - E(Y|X) (E(Y|X) - g(X))|X\right] \} = E \{(E(Y|X) - g(X)) \underbrace{E \left[(Y - E(Y|X) |X\right]}_{=0} \} = 0 $$ Where the term $E(E(Y|X) - g(X))$ goes out the conditional expectation to $X$ because it is a function of $X$.

$\endgroup$
2
  • $\begingroup$ How do you show that the middle term is zero? $\endgroup$
    – Zhanxiong
    Dec 27, 2017 at 19:20
  • $\begingroup$ @Zhanxiong It is already hinted that you have to condtiion to X. Because the second factor is a function of $X$ it comes out of the conditioning and the first factpr is is equal to zero. $\endgroup$
    – Manuel
    Dec 28, 2017 at 16:05
2
$\begingroup$

Since answers have already been provided, here is a short intro to conditional expectations as $L^2$ projections, i.e. for an alternative path to conditional expectations in which the question asked is taken more or less as definition.

Intuitively, taking the expected value of $Y$ conditioned on $X$ means averaging $Y$ over events which produce the outcomes of $X$. If we think of the $\sigma$-algebra generated by $X$, $\sigma(X)$, as grouping events leading to the same outcomes of $X$, then we see that we want the conditional expectation $\mathbb{E}[Y|X]$ to be a r.v. over $\sigma(x)$, closest in some sense to the original $Y$.

Therefore, if we work in $L^2(\Omega, \mathcal{A}, P)$, with $\mathcal{A}$ a $\sigma$-algebra over $\Omega$, then the conditional expectation $\mathbb{E}[Y|X]$ has a natural definition as the $L^2$ projection of Y onto $L^2$ of $\sigma(X)$ since $L^2(\Omega, \sigma(X), P)$ is a closed subspace of $L^2(\Omega, \mathcal{A}, P)$:

$$\mathbb{E}[Y|X] = \operatorname{argmin}_g \mathbb{E}[(Y-g(X))^2]$$

over all Borel $g$ such that $g(X) \in L^2$ (recall that $Z$ is measurable wrt. $\sigma(X)$ iff there is some Borel $g$ such that $Z=g(X)$)

The case $L^1$ is not so straightforward because we lack the Hilbert space structure, see e.g. "Jacod, Protter: Probability essentials", Chapter 23.

Conditional expectations can be taken over any sub $\sigma$-algebra of $\mathcal{A}$. For a nice introduction starting from conditioning on $\sigma$-algebras and with pictures and intuition, see "Conditional Expectations without tears".

$\endgroup$
1
  • $\begingroup$ Nice intro and reference! $\endgroup$
    – Manuel
    Dec 28, 2017 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.