# Problem with proof of Conditional expectation as best predictor

I have an issue with the proof of

$$E(Y|X) \in \arg \min_{g(X)} E\Big[\big(Y - g(X)\big)^2\Big]$$

which very likely reveal a deeper misunderstanding of expectations and conditional expectations.

The proof I know goes as follows ( another version of this proof can be found here)

\begin{align*} &\arg \min_{g(X)} E\Big[\big(Y - g(x)\big)^2\Big]\\ = &\arg \min_{g(X)} E \Big[ \big(Y - E(Y|X) + E(Y|X) - g(X)\big)^2\Big]\\ =&\arg \min_{g(x)} E \Big[ \big(Y - E(Y|X)\big)^2 + 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big]\\ =&\arg \min_{g(x)} E \Big[ 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big]\\ \end{align*}

The proof then typically continues with an argument showing that $$2 E\Big[ \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big)\Big] = 0$$, and hence

\begin{align*} \arg \min_{g(x)} E\Big[\big(Y - g(x)\big)^2\Big] = \arg \min_{g(x)} E \Big[\big(E(Y|X) - g(X)\big)^2\Big] \end{align*}

which can be seen to be minimized when $$g(X) = E(Y|X)$$.

My puzzles about the proof are the following:

1. Consider

$$E \Big[ 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big]$$.

It seems to me that, independently of any argument showing that the first term is always equal to zero, one can see that setting $$g(X) = E(Y|X)$$ minimizes the expression as it implies $$\big(E(Y|X) - g(X)\big) =0$$ and hence

$$E \Big[ 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big] = E( 0 + 0)$$ = 0.

But if this is true, then one might repeat the proof replacing $$E(Y|X)$$ by any other function of $$X$$, say $$h(X)$$, and get to the conclusion that it is $$h(X)$$ that minimizes the expression. So there must be something I misunderstand (right?).

1. I have some doubts about the meaning of $$E[(Y−g(X))^2]$$ in the statement of the problem. How should the notation be interpreted? Does it mean

$$E_X[(Y−g(X))^2]$$, $$E_Y[(Y−g(X))^2]$$ or $$E_{XY}[(Y−g(X))^2]$$?

(This is an adaptation from Granger & Newbold(1986) "Forecasting Economic Time Series").

By construction, your error cost function is $\left[Y-g(X)\right]^2$. This incorporates a critical assumption (that the error cost function is symmetric around zero) -a different error cost function would not necessarily have the conditional expected value as the $\arg \min$ of its expected value. You cannot minimize your error cost function because it contains unknown quantities. So you decide to minimize its expected value instead. Then your objective function becomes

$$E\left[Y-g(X)\right]^2 = \int_{-\infty}^{\infty}\left[y-g(X)\right]^2f_{Y|X}(y|x)dy$$

which I believe answers also your second question. It is intuitive that the expected value will be of $Y$ conditional on $X$, since we are trying to estimate/forecast $Y$ based on $X$. Decompose the square to obtain

$$E\left[Y-g(X)\right]^2 = \int_{-\infty}^{\infty}y^2f_{Y|X}(y|x)dy -2g(X)\int_{-\infty}^{\infty}yf_{Y|X}(y|x)dy \\+ \Big[g(X)\Big]^2\int_{-\infty}^{\infty}f_{Y|X}(y|x)dy$$

The first term does not contain $g(X)$ so it does not affect minimization, and it can be ignored. The integral in the second term equals the conditional expected value of $Y$ given $X$, and the integral in the last term equals unity. So

$$\arg \min_{g(x)} E\left[Y-g(X)\right]^2 = \arg \min_{g(x)} \Big\{ -2g(X)E(Y\mid X) + \Big[g(X)\Big]^2 \Big\}$$

The first derivative w.r.t $g(X)$ is $-2E(Y\mid X) + 2g(X)$ leading to the first order condition for minimization $g(X) = E(Y\mid X)$ while the second derivative is equal to $2>0$ which is sufficient for a minimum.

The OP is puzzled by the approach stated in the question, because it seems tautological. It isn't, because while using the tactic of adding and subtracting makes a specific part of the objective function zero for an arbitrary choice of the term that is added and subtracted, it does NOT equalize the value function , namely the value of the objective function evaluated at the candidate minimizer.

For the choice $g(X) = E(Y \mid X)$ we have the value function $V\left(E(Y\mid X)\right) = E\Big[ (Y-E(Y \mid X))^2\mid X\Big]$ For the arbitrary choice $g(X) = h(X)$we have the value funtion $V\left(h(X)\right) = E\Big[ (Y-h(X))^2\mid X\Big]$.

I claim that

$$V\left(E(Y\mid X)\right) \le V\left(h(X)\right)$$ $$\Rightarrow E(Y^2\mid X) -2E\Big [(YE(Y \mid X))\mid X\Big] + E\Big [(E(Y \mid X))^2\mid X\Big] \\\le E(Y^2\mid X) -2E\Big [(Yh(X))\mid X\Big] + E\Big [(h(X))^2\mid X\Big]$$

The first term of the LHS and the RHS cancel out. Also note that the outer expectation is conditional on $X$. By the properties of conditional expectations we end up with

$$...\Rightarrow -2E(Y \mid X)\cdot E\Big (Y\mid X\Big) + \Big [E(Y \mid X)\Big]^2 \le -2E(Y\mid X)h(X) + \Big [h(X)\Big]^2$$

$$\Rightarrow 0 \le \Big [E(Y \mid X)\Big]^2-2E(Y\mid X)h(X) + \Big [h(X)\Big]^2$$

$$\Rightarrow 0 \le \Big [E(Y \mid X) - h(x)\Big]^2$$ which holds with strict inequality if $h(x) \neq E(Y \mid X)$. So $E(Y \mid X)$ is the global and unique minimizer.

But this also says that the "add-and-subtract" approach is not the most illuminating way of proof here.

• Thanks for you answer. It helps clarifying my second question. As I tried to convey in the title of the question, my main issue (the first one in the post) was more about the proof mechanism. My main concern is about my understanding of the proof I presented in the question. As I explained, my understanding of the proof leads me to blatantly problematic statement. So I would like to understand were my mistake is as it might reveal some deeper misunderstandings about concepts of expectaction and conditional expectation. Any thoughts about this? Oct 4, 2013 at 4:00
• I added some explanation on the "add and subtract" approach to proof. Oct 4, 2013 at 10:27
• Took me some time to understand it, but I finally got my basic mistake : true enough $E \Big[ - 2 \big(Y - h(X) \big) \big(h(X) - g(X)\big) + \big(h(X) - g(X)\big)^2\Big] = 0$ when $g(X) = h(X)$, but by no means does it imply that $h(X)$ minimizes the expression. There is no reason which the bracketed expression could not be lower than zero. Because of the minus sign in front of $\big(Y - h(X) \big) \big(h(X) - g(X)\big)$ one could find some $g(X)$ such that $E \Big[ - 2 \big(Y - h(X) \big) \big(h(X) - g(X)\big) + \big(h(X) - g(X)\big)^2\Big] < 0$. Oct 4, 2013 at 16:24
• Hmmm... the minus sign in the expression you refer to is a mistake - it should be a plus sign. You could of course then rearrange the terms to obtain again a minus sign... does this hurt the intuition you gained? Oct 4, 2013 at 21:03
• Thanks for keeping up with the question. I edited the initial post to correct for this mistake. Fortunately, I think it does not hurt the gained intuition. Actually it helps me understand yet another mistake : I was assuming that the minus sign was important to guarantee that $0$ was not necessarily the minimum of $E[−2(Y−h(X))(h(X)−g(X))+(h(X)−g(X))^2]$. But I realize this is not just about the sign before the 2. (Hopefully) What I really needed to understand is that, in general (i.e. for arbitrary $h(X)$) $E[2(Y−h(X))(h(X)−g(X))]$ needs not be minimized when $g(X)=h(X)$ (right?). Oct 4, 2013 at 23:36

There's a mathematical point of view that is very simple. What you have is a projection problem in a Hilbert space, much like projecting a vector in $$\mathbb{R}^n$$ onto a subspace.

Let $$(\Omega, \mathcal{F}, \mu)$$ denote the underlying probability space. For the problem to make sense, consider the random variables with finite second moments, that is, the Hilbert space $$L^2(\Omega, \mathcal{F}, \mu)$$. The problem now is this: given $$X, Y \in L^2(\Omega, \mathcal{F}, \mu)$$, find the projection of $$Y$$ onto the subspace $$L^2(\Omega, \mathcal{F}_X, \mu)$$, where $$\mathcal{F}_X$$ is the $$\sigma$$-subalgebra of $$\mathcal{F}$$ generated by $$X$$. (Just as in the finite dimensional case, minimizing $$L^2$$-distance to a subspace means finding the projection). The desired projection is $$E(Y|X)$$, by construction. (This actually characterizes $$E(Y|X)$$, if one inspects the proof of existence).

Edit

Re "...,by construction."

By definition, the conditional mean of $$Y$$ on $$X$$ is a random variable $$\psi$$ with the following two properties:

1. $$\psi$$ lies in $$L^2(\Omega, \mathcal{F}_X, \mu)$$.

2. $$E[\psi 1_{A}] = E[Y 1_{A}]$$, for all $$A \in \mathcal{F}_X$$, which implies that $$E[\psi g] = E[Y g]$$, for all $$g \in L^2(\Omega, \mathcal{F}_X, \mu)$$, by standard argument use denseness of simple functions.

Standard Hilbert space projection arguments show that such a $$\psi$$ always exists and is unique.

This applies to any Hilbert space. The above can be re-phrased verbatim for, say, $$\mathbb{R}^n$$:

Let $$Y \in \mathbb{R}^n$$ and $$V$$ be a subspace. Then the projection of $$Y$$ onto $$V$$ is characterized by the same two properties:

1. $$\psi$$ lies in $$V$$.

2. $$\langle \psi, g \rangle = \langle Y, g \rangle$$, for all $$g \in V$$.

Note This discussion is restricted to $$L^2$$ random variables, as the original question does implicitly. Conditional mean in general is defined for $$L^1$$ random variables, which is a larger than $$L^2$$. $$L^1$$ is a Banach space and conditional mean is still a projection, in an appropriate sense.

• This is a beautiful response.
– jII
Mar 18, 2015 at 20:36
• This is a very elegant proof, though I do not have a very good understanding of probability theory. It would be great if you can elaborate on your answer. The last part was not very clear where you mentioned: "by construction". In particular, how to find the projection on $L^2(\Omega,\mathcal{F},\mu)$? Mar 13, 2020 at 4:31
• Thanks @Michael for the details. I really appreciate it! Mar 14, 2020 at 6:54

Note that to prove the answer, you really only need to show that

$$E \Big[ -2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) \Big] = 0$$

As for which expectation to take, you take it conditionally, otherwise the term

$$\arg \min_{g(X)} E\Big[\big(Y - g(X)\big)^2\Big]$$

Doesn't make sense, as $g(X)$ is a random variable if $E$ is $E_{XY}$ and not $E_{Y|X}$. Show you should really write $E\Big[\big(Y - g(X)\big)^2|X\Big]$ or $E_{Y|X}\Big[\big(Y - g(X)\big)^2\Big]$ to make this clear. Now given this clarification, the term $\big(E(Y|X) - g(X)\big)$ is a constant, and can be pulled outside the expecation, and you have:

$$-2\big(E(Y|X) - g(X)\big)E \Big[ \big(Y - E(Y|X)\big)|X\Big]=-2\big(E(Y|X) - g(X)\big)\Big[ E(Y|X) - E\big[E(Y|X)|X\big]\Big]=-2\big(E(Y|X) - g(X)\big)\Big[ E(Y|X) - E(Y|X)\Big]=0$$

Hence you can write the objective function as:

$$E_{Y|X}\Big[\big(Y - g(X)\big)^2\Big]=E_{Y|X}\Big[\big(Y - E_{Y|X}(Y|X)\big)^2\Big]+\big(E_{Y|X}(Y|X) - g(X)\big)^2$$

The minimiser is obvious from here. Note that if you were to average over $X$ as well, then a very similar argument can be used to show:

$$E_{X}\Big[\big(E(Y|X) - g(X)\big)^2\Big]=E_{X}\Big[\big(E_{Y|X}(Y|X) - E_X\big[E_{Y|X}(Y|X)\big]\big)^2\Big]+\Big(E_{X}\big[E_{Y|X}(Y|X)\big] - E_X\big[g(X)\big]\Big)^2$$

This shows that if you set $g(X)=E_{Y|X}(Y|X)$ for each $X$, then you also have a minimiser over this function as well. So in some sense it doesn't really matter whether $E$ is $E_{YX}$ or $E_{Y|X}$.

Regarding your last question, the expectation can be either w.r.t. $$p(x,y)$$ (the unconditional error) or w.r.t. $$p(y\mid x)$$ (the conditional error at each value $$X = x$$). Happily, minimizing the conditional error at each value $$X = x$$ also minimizes the unconditional error, so this is not a crucial distinction.