I try to minimize mean squared error function defined as:

$E\left[Y - f(X)\right]^2$

I summarized the minimization procedure from different online sources (e.g., URL 1 (p. 4), URL 2 (p. 8)) in the following lines.

First add and subtract $E[Y | X]$:

$E\left[\left\lbrace(Y - E[Y | X]) - (f(X) - E[Y|X])\right\rbrace^2\right]$

Expanding the quadratic yield:

$E\left[\left(Y - E[Y|X]\right)^2 + \left(f(X) - E[Y|X]\right)^2 - 2 \left(Y - E[Y|X]\right)\left(f(X) - E[Y|X]\right)\right]$

First term is not affected by the choice of $f(X)$; third term is $0$, so the whole expression is minimized if $f(X) = E(Y|X)$.

Question 1: I wonder what is the motivation to add and subtract $E[Y | X]$ in the first step of the procedure?

Question 2: How to explain in plain English why third term in quadratic is $0$?


Concerning your first question, adding and subtracting is a trick in statistics which is often used to more easily work with certain expressions. By adding and subtracting you do not change your equation but it makes it possible to group certain terms to obtain the result more easily.

For your second question, to make this point more formally, we want to show the conditional expectation function (CEF) prediction property: $$E(Y|X) = \text{arg min}_{f(X)} E[(Y-f(X))^2]$$ I guess that not stating that $f(X)$ is the minimization argument in the question caused confusion for some. The CEF also has the following decomposition property: $$Y=E(Y|X) + \epsilon $$ where $\epsilon $ is a random variable such that $E(\epsilon|X) =0$ and $E(h(X)\epsilon)=0$.

In your last expression you have $(Y-E(Y|X)) = \epsilon$ and $(f(X)-E(Y|X)) = h(X)$ is a function of $X$. Then you use the previous property of $\epsilon$ to show that $-2E[h(X)\epsilon]=0$, hence the last expression is zero. This proof goes by using properties of the CEF rather than anything unnecessarily complicated - so it's plain English for most parts.

  • $\begingroup$ Your second point is wrong. You want to show $f(X)=E(Y|X)$, and so you cannot assume it! $\endgroup$ – M Turgeon May 3 '14 at 19:50
  • $\begingroup$ The function $f(X)$ can be anything. The point of the proof is to show that the MSE is minimized by the conditional mean. This is to set the stage for relating the conditional mean to regression (see URL 1 in Andrej's post). $\endgroup$ – Andy May 3 '14 at 19:55
  • $\begingroup$ My point is that you are not answering the second part of the question when you say "So all terms where you have $f(X)-E(Y|X)$ are zero". This is both false and misleading. The fact that it is zero is a consequence of the fact that they are equal, and therefore you cannot use this to prove they are indeed equal. $\endgroup$ – M Turgeon May 3 '14 at 20:00
  • 5
    $\begingroup$ If you quote me then please quote completely: $f(X) - E(Y|X)$ is zero for the choice $f(X) = E(Y|X)$. This is known as the CEF prediction property and in class you usually show it to motivate least squares as projection of $Y$ on $X$. See URL 1 or any other econometrics lecture on this topic for that matter. $\endgroup$ – Andy May 3 '14 at 20:16
  • 1
    $\begingroup$ I don't think that this is quite right because if the last term isn't always zero, but zero because you happen to pick a value that makes it zero, it might be possible to make it negative and then you'd have to worry about the interplay between it and the second term. $\endgroup$ – testuser May 4 '14 at 3:49

For your second question, you want to show that $$E\left[ (Y-E(Y|X)(f(X)-E(Y|X))\right]=0.$$ Now, if we look at the first term of the product, if we didn't have a conditional expectation, we would have $$E(Y-E(Y))=E(Y)-E(Y)=0.$$ But by the Law of Total expectation, we know that $$E(W)=E(E(W|Z)),$$ so you can actually write $$E(Y-E(Y|X)) = E(E(Y-E(Y|X)|X)) = E(E(Y|X)-E(Y|X)) =E(0)=0.$$ To finish the proof, note that conditional on $X$, the second term is a constant, and therefore the expectation of the product is the product of the expectations: $$E\left[ (Y-E(Y|X))(f(X)-E(Y|X))|X\right]=E\left[ (Y-E(Y|X))|X\right]\cdot E\left[(f(X)-E(Y|X))|X\right]$$

  • $\begingroup$ Could you explain the second step of the equation following "so you can actually write" in more detail (e.g. after the second =)? how does one derive the first term of E(Y|X)? $\endgroup$ – user1885116 May 3 '14 at 20:31
  • 2
    $\begingroup$ I understand the first part of your answer where using the Law of total expectation you show that $E(E(Y|X) - E(Y|X)) = 0$. This way the expression $2 (Y - E[Y|X])(f(X) - E[Y|X]) = 0$, so could you please elaborate the second part of your answer, following To finish the proof... $\endgroup$ – Andrej May 4 '14 at 5:20
  • $\begingroup$ @user1885116 I simply apply the law of total expectation to $Y-E(Y|X)$. The inner expectation is conditional on $X$, and therefore $E(Y|X)$ is treated as a constant. This therefore gives $$E(Y-E(Y|X)|X)=E(Y|X)-E(E(Y|X)|X)=E(Y|X)-E(Y|X).$$ $\endgroup$ – M Turgeon May 4 '14 at 20:57
  • $\begingroup$ @Andrej My last comment is about the fact that in general, the expectation of a product is not the product of expectations, but that in this case it is so the argument goes through. $\endgroup$ – M Turgeon May 4 '14 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.