Problem with proof of Conditional expectation as best predictor

Question

I have an issue with the proof of

$E(Y|X) \in \arg \min_{g(X)} E\Big[\big(Y - g(X)\big)^2\Big]$

which very likely reveal a deeper misunderstanding of expectations and conditional expectations.

The proof I know goes as follows ( another version of this proof can be found here)

\begin{align*} &\arg \min_{g(X)} E\Big[\big(Y - g(x)\big)^2\Big]\\ = &\arg \min_{g(X)} E \Big[ \big(Y - E(Y|X) + E(Y|X) - g(X)\big)^2\Big]\\ =&\arg \min_{g(x)} E \Big[ \big(Y - E(Y|X)\big)^2 + 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big]\\ =&\arg \min_{g(x)} E \Big[ 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big]\\ \end{align*}

The proof then typically continues with an argument showing that $2 E\Big[ \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big)\Big] = 0$, and hence

\begin{align*} \arg \min_{g(x)} E\Big[\big(Y - g(x)\big)^2\Big] = \arg \min_{g(x)} E \Big[\big(E(Y|X) - g(X)\big)^2\Big] \end{align*}

which can be seen to be minimized when $g(X) = E(Y|X)$.

My puzzles about the proof are the following:

1. Consider

$E \Big[ 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big]$.

It seems to me that, independently of any argument showing that the first term is always equal to zero, one can see that setting $g(X) = E(Y|X)$ minimizes the expression as it implies $\big(E(Y|X) - g(X)\big) =0$ and hence

$E \Big[ 2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) + \big(E(Y|X) - g(X)\big)^2\Big] = E( 0 + 0)$ = 0.

But if this is true, then one might repeat the proof replacing $E(Y|X)$ by any other function of $X$, say $h(X)$, and get to the conclusion that it is $h(X)$ that minimizes the expression. So there must be something I misunderstand (right?).

2. I have some doubts about the meaning of $E[(Y−g(X))^2]$ in the statement of the problem. How should the notation be interpreted? Does it mean

$E_X[(Y−g(X))^2]$, $E_Y[(Y−g(X))^2]$ or $E_{XY}[(Y−g(X))^2]$?

Answer 1

(This is an adaptation from Granger & Newbold(1986) "Forecasting Economic Time Series").

By construction, your error cost function is $\left[Y-g(X)\right]^2$. This incorporates a critical assumption (that the error cost function is symmetric around zero) -a different error cost function would not necessarily have the conditional expected value as the $\arg \min$ of its expected value. You cannot minimize your error cost function because it contains unknown quantities. So you decide to minimize its expected value instead. Then your objective function becomes

$$E\left[Y-g(X)\right]^2 = \int_{-\infty}^{\infty}\left[y-g(X)\right]^2f_{Y|X}(y|x)dy $$

which I believe answers also your second question. It is intuitive that the expected value will be of $Y$ conditional on $X$, since we are trying to estimate/forecast $Y$ based on $X$. Decompose the square to obtain

$$E\left[Y-g(X)\right]^2 = \int_{-\infty}^{\infty}y^2f_{Y|X}(y|x)dy -2g(X)\int_{-\infty}^{\infty}yf_{Y|X}(y|x)dy \\+ \Big[g(X)\Big]^2\int_{-\infty}^{\infty}f_{Y|X}(y|x)dy$$

The first term does not contain $g(X)$ so it does not affect minimization, and it can be ignored. The integral in the second term equals the conditional expected value of $Y$ given $X$, and the integral in the last term equals unity. So

$$\arg \min_{g(x)} E\left[Y-g(X)\right]^2 = \arg \min_{g(x)} \Big\{ -2g(X)E(Y\mid X) + \Big[g(X)\Big]^2 \Big\}$$

The first derivative w.r.t $g(X)$ is $-2E(Y\mid X) + 2g(X)$ leading to the first order condition for minimization $g(X) = E(Y\mid X)$ while the second derivative is equal to $2>0$ which is sufficient for a minimum.

ADDENDUM:The logic of the "add and subtract" proof approach.

The OP is puzzled by the approach stated in the question, because it seems tautological. It isn't, because while using the tactic of adding and subtracting makes a specific part of the objective function zero for an arbitrary choice of the term that is added and subtracted, it does NOT equalize the value function , namely the value of the objective function evaluated at the candidate minimizer.

For the choice $g(X) = E(Y \mid X)$ we have the value function $ V\left(E(Y\mid X)\right) = E\Big[ (Y-E(Y \mid X))^2\mid X\Big]$ For the arbitrary choice $g(X) = h(X)$we have the value funtion $ V\left(h(X)\right) = E\Big[ (Y-h(X))^2\mid X\Big]$.

I claim that

$$V\left(E(Y\mid X)\right) \le V\left(h(X)\right)$$ $$\Rightarrow E(Y^2\mid X) -2E\Big [(YE(Y \mid X))\mid X\Big] + E\Big [(E(Y \mid X))^2\mid X\Big] \\\le E(Y^2\mid X) -2E\Big [(Yh(X))\mid X\Big] + E\Big [(h(X))^2\mid X\Big]$$

The first term of the LHS and the RHS cancel out. Also note that the outer expectation is conditional on $X$. By the properties of conditional expectations we end up with

$$...\Rightarrow -2E(Y \mid X)\cdot E\Big (Y\mid X\Big) + \Big [E(Y \mid X)\Big]^2 \le -2E(Y\mid X)h(X) + \Big [h(X)\Big]^2$$

$$\Rightarrow 0 \le \Big [E(Y \mid X)\Big]^2-2E(Y\mid X)h(X) + \Big [h(X)\Big]^2$$

$$\Rightarrow 0 \le \Big [E(Y \mid X) - h(x)\Big]^2$$ which holds with strict inequality if $h(x) \neq E(Y \mid X)$. So $E(Y \mid X)$ is the global and unique minimizer.

But this also says that the "add-and-subtract" approach is not the most illuminating way of proof here.

Answer 2

Note that to prove the answer, you really only need to show that

$$E \Big[ -2 \big(Y - E(Y|X)\big) \big(E(Y|X) - g(X)\big) \Big] = 0$$

As for which expectation to take, you take it conditionally, otherwise the term

$$\arg \min_{g(X)} E\Big[\big(Y - g(X)\big)^2\Big]$$

Doesn't make sense, as $g(X)$ is a random variable if $E$ is $E_{XY}$ and not $E_{Y|X}$. Show you should really write $E\Big[\big(Y - g(X)\big)^2|X\Big]$ or $E_{Y|X}\Big[\big(Y - g(X)\big)^2\Big]$ to make this clear. Now given this clarification, the term $\big(E(Y|X) - g(X)\big)$ is a constant, and can be pulled outside the expecation, and you have:

$$-2\big(E(Y|X) - g(X)\big)E \Big[ \big(Y - E(Y|X)\big)|X\Big]=-2\big(E(Y|X) - g(X)\big)\Big[ E(Y|X) - E\big[E(Y|X)|X\big]\Big]=-2\big(E(Y|X) - g(X)\big)\Big[ E(Y|X) - E(Y|X)\Big]=0$$

Hence you can write the objective function as:

$$E_{Y|X}\Big[\big(Y - g(X)\big)^2\Big]=E_{Y|X}\Big[\big(Y - E_{Y|X}(Y|X)\big)^2\Big]+\big(E_{Y|X}(Y|X) - g(X)\big)^2$$

The minimiser is obvious from here. Note that if you were to average over $X$ as well, then a very similar argument can be used to show:

$$E_{X}\Big[\big(E(Y|X) - g(X)\big)^2\Big]=E_{X}\Big[\big(E_{Y|X}(Y|X) - E_X\big[E_{Y|X}(Y|X)\big]\big)^2\Big]+\Big(E_{X}\big[E_{Y|X}(Y|X)\big] - E_X\big[g(X)\big]\Big)^2$$

This shows that if you set $g(X)=E_{Y|X}(Y|X)$ for each $X$, then you also have a minimiser over this function as well. So in some sense it doesn't really matter whether $E$ is $E_{YX}$ or $E_{Y|X}$.

Answer 3

There's a mathematical point of view that is very simple. What you have is a projection problem in a Hilbert space, much like projecting a vector in $\mathbb{R}^n$ onto a subspace.

Let $(\Omega, \mathcal{F}, \mu)$ denote the underlying probability space. For the problem to make sense, consider the random variables with finite second moments, that is, the Hilbert space $L^2(\Omega, \mathcal{F}, \mu)$. The problem now is this: given $X, Y \in L^2(\Omega, \mathcal{F}, \mu)$, find the projection of $Y$ onto the subspace $L^2(\Omega, \mathcal{F}_X, \mu)$, where $\mathcal{F}_X$ is the $\sigma$-subalgebra of $\mathcal{F}$ generated by $X$. (Just as in the finite dimensional case, minimizing $L^2$-distance to a subspace means finding the projection). The desired projection is $E(Y|X)$, by construction. (This actually characterizes $E(Y|X)$, if one inspects the proof of existence).

Edit

Re "...,by construction."

By definition, the conditional mean of $Y$ on $X$ is a random variable $\psi$ with the following two properties:

1. $\psi$ lies in $L^2(\Omega, \mathcal{F}_X, \mu)$.

2. $E[\psi 1_{A}] = E[Y 1_{A}]$, for all $A \in \mathcal{F}_X$, which implies that $E[\psi g] = E[Y g]$, for all $g \in L^2(\Omega, \mathcal{F}_X, \mu)$, by standard argument use denseness of simple functions.

Standard Hilbert space projection arguments show that such a $\psi$ always exists and is unique.

This applies to any Hilbert space. The above can be re-phrased verbatim for, say, $\mathbb{R}^n$:

Let $Y \in \mathbb{R}^n$ and $V$ be a subspace. Then the projection of $Y$ onto $V$ is characterized by the same two properties:

1. $\psi$ lies in $V$.

2. $\langle \psi, g \rangle = \langle Y, g \rangle$, for all $g \in V$.

Note This discussion is restricted to $L^2$ random variables, as the original question does implicitly. Conditional mean in general is defined for $L^1$ random variables, which is a larger than $L^2$. $L^1$ is a Banach space and conditional mean is still a projection, in an appropriate sense.

Answer 4

Regarding your last question, the expectation can be either w.r.t. $p(x,y)$ (the unconditional error) or w.r.t. $p(y\mid x)$ (the conditional error at each value $X = x$). Happily, minimizing the conditional error at each value $X = x$ also minimizes the unconditional error, so this is not a crucial distinction.