8
$\begingroup$

I try to minimize mean squared error function defined as:

$E\left[Y - f(X)\right]^2$

I summarized the minimization procedure from different online sources (e.g., URL 1 (p. 4), URL 2 (p. 8)) in the following lines.

First add and subtract $E[Y | X]$:

$E\left[\left\lbrace(Y - E[Y | X]) - (f(X) - E[Y|X])\right\rbrace^2\right]$

Expanding the quadratic yield:

$E\left[\left(Y - E[Y|X]\right)^2 + \left(f(X) - E[Y|X]\right)^2 - 2 \left(Y - E[Y|X]\right)\left(f(X) - E[Y|X]\right)\right]$

First term is not affected by the choice of $f(X)$; third term is $0$, so the whole expression is minimized if $f(X) = E(Y|X)$.

Question 1: I wonder what is the motivation to add and subtract $E[Y | X]$ in the first step of the procedure?

Question 2: How to explain in plain English why third term in quadratic is $0$?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
6
$\begingroup$

Concerning your first question, adding and subtracting is a trick in statistics which is often used to more easily work with certain expressions. By adding and subtracting you do not change your equation but it makes it possible to group certain terms to obtain the result more easily.

For your second question, to make this point more formally, we want to show the conditional expectation function (CEF) prediction property: $$E(Y|X) = \text{arg min}_{f(X)} E[(Y-f(X))^2]$$ I guess that not stating that $f(X)$ is the minimization argument in the question caused confusion for some. The CEF also has the following decomposition property: $$Y=E(Y|X) + \epsilon $$ where $\epsilon $ is a random variable such that $E(\epsilon|X) =0$ and $E(h(X)\epsilon)=0$.

In your last expression you have $(Y-E(Y|X)) = \epsilon$ and $(f(X)-E(Y|X)) = h(X)$ is a function of $X$. Then you use the previous property of $\epsilon$ to show that $-2E[h(X)\epsilon]=0$, hence the last expression is zero. This proof goes by using properties of the CEF rather than anything unnecessarily complicated - so it's plain English for most parts.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Your second point is wrong. You want to show $f(X)=E(Y|X)$, and so you cannot assume it! $\endgroup$
    – M Turgeon
    Commented May 3, 2014 at 19:50
  • $\begingroup$ The function $f(X)$ can be anything. The point of the proof is to show that the MSE is minimized by the conditional mean. This is to set the stage for relating the conditional mean to regression (see URL 1 in Andrej's post). $\endgroup$
    – Andy
    Commented May 3, 2014 at 19:55
  • $\begingroup$ My point is that you are not answering the second part of the question when you say "So all terms where you have $f(X)-E(Y|X)$ are zero". This is both false and misleading. The fact that it is zero is a consequence of the fact that they are equal, and therefore you cannot use this to prove they are indeed equal. $\endgroup$
    – M Turgeon
    Commented May 3, 2014 at 20:00
  • 6
    $\begingroup$ If you quote me then please quote completely: $f(X) - E(Y|X)$ is zero for the choice $f(X) = E(Y|X)$. This is known as the CEF prediction property and in class you usually show it to motivate least squares as projection of $Y$ on $X$. See URL 1 or any other econometrics lecture on this topic for that matter. $\endgroup$
    – Andy
    Commented May 3, 2014 at 20:16
  • 1
    $\begingroup$ I don't think that this is quite right because if the last term isn't always zero, but zero because you happen to pick a value that makes it zero, it might be possible to make it negative and then you'd have to worry about the interplay between it and the second term. $\endgroup$
    – user27028
    Commented May 4, 2014 at 3:49
2
$\begingroup$

For your second question, you want to show that $$E\left[ (Y-E(Y|X)(f(X)-E(Y|X))\right]=0.$$ Now, if we look at the first term of the product, if we didn't have a conditional expectation, we would have $$E(Y-E(Y))=E(Y)-E(Y)=0.$$ But by the Law of Total expectation, we know that $$E(W)=E(E(W|Z)),$$ so you can actually write $$E(Y-E(Y|X)) = E(E(Y-E(Y|X)|X)) = E(E(Y|X)-E(Y|X)) =E(0)=0.$$ To finish the proof, note that conditional on $X$, the second term is a constant, and therefore the expectation of the product is the product of the expectations: $$E\left[ (Y-E(Y|X))(f(X)-E(Y|X))|X\right]=E\left[ (Y-E(Y|X))|X\right]\cdot E\left[(f(X)-E(Y|X))|X\right]$$

Improve this answer
$\endgroup$
4
  • $\begingroup$ Could you explain the second step of the equation following "so you can actually write" in more detail (e.g. after the second =)? how does one derive the first term of E(Y|X)? $\endgroup$
    – user1885116
    Commented May 3, 2014 at 20:31
  • 2
    $\begingroup$ I understand the first part of your answer where using the Law of total expectation you show that $E(E(Y|X) - E(Y|X)) = 0$. This way the expression $2 (Y - E[Y|X])(f(X) - E[Y|X]) = 0$, so could you please elaborate the second part of your answer, following To finish the proof... $\endgroup$
    – Andrej
    Commented May 4, 2014 at 5:20
  • $\begingroup$ @user1885116 I simply apply the law of total expectation to $Y-E(Y|X)$. The inner expectation is conditional on $X$, and therefore $E(Y|X)$ is treated as a constant. This therefore gives $$E(Y-E(Y|X)|X)=E(Y|X)-E(E(Y|X)|X)=E(Y|X)-E(Y|X).$$ $\endgroup$
    – M Turgeon
    Commented May 4, 2014 at 20:57
  • $\begingroup$ @Andrej My last comment is about the fact that in general, the expectation of a product is not the product of expectations, but that in this case it is so the argument goes through. $\endgroup$
    – M Turgeon
    Commented May 4, 2014 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.