2
$\begingroup$

This is taken from the book "A First Course in Probability" from Sheldon Ross:

Sometimes a situation arises in which the value of a random variable $X$ is observed and then, on the basis of the observed value, an attempt is made to predict the value of a second random variable $Y$. Let $g(X)$ denote the predictor; that is, if $X$ is observed to equal $x$, then $g(x)$ is our prediction for the value of $Y$. Clearly, we would like to choose $g$ so that $g(X)$ tends to be close to $Y$. One possible criterion for closeness is to choose $g$ so as to minimize $E[(Y − g(X))^{2}]$. We now show that, under this criterion, the best possible predictor of $Y$ is $g(X) = E[Y|X]$.

So, I understand most of the proof that is provided in the book, but the start of the proof is what I do not understand: enter image description here

So, why is ok to write the conditional expectation? We are trying to minimize the unconditional expectation, why can we assume that is the same as trying to minimize the conditional expectation on $X$?

$\endgroup$
2
$\begingroup$

I guess you want to understand why he began the proof with $\mathbb{E}[(Y-g(X))^2|X]$ instead of $\mathbb{E}[(Y-g(X))^2]$. By the Law of Iterated Expectations we have that

$$ \mathbb{E}[(Y-g(X))^2] = \mathbb{E}[\mathbb{E}[(Y-g(X))^2|X]]\quad.$$

So, if you want to minimize the first quantity, what you can do is, for each $X$, minimize $\mathbb{E}[(Y-g(X))^2|X]$. To be more rigorous, suppose that there is a function f such that

$$ \mathbb{E}[(Y-g(X))^2|X] \geq \mathbb{E}[(Y-f(X))^2|X]\quad.$$

Consequently, using the properties of expectation, it follows that

$$ \mathbb{E}[(Y-g(X))^2] = \mathbb{E}[\mathbb{E}[(Y-g(X))^2|X]] \geq \mathbb{E}[\mathbb{E}[(Y-f(X))^2|X]] = \mathbb{E}[(Y-f(X))^2] \quad.$$

This proves rigorously why you can focus on minimizing the conditional expectation.

$\endgroup$
3
$\begingroup$

In the end, the proof states the following: $$E[(Y-g(X))^2|X]=E[(Y-E[Y|X]^2)|X]+E[(E[Y|X]-g(X))^2|X]$$ If you take the expectation of both sides and then use the law of iterated expectations, i.e. $E[A]=E[E[A|B]]$, we have what we need: $$E[(Y-g(X))^2]=E[(Y-E[Y|X]^2)]+E[(E[Y|X]-g(X))^2]$$

$\endgroup$
1
  • $\begingroup$ Good answer (+1), but I think there is a more straightforward way to see this. The argument makes it look like we did a bunch of calculations and stumbled on the answer. If we focus on writing the desired quantity in terms of the conditional expectation from the very beginning, it is clear that we could tackle on minimizing $\mathbb{E}[(Y-g(X))^2|X]$. $\endgroup$ Oct 16 '19 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.