# Confusion in regression function derivation

I was reading 'The Elements of Statistical Learning' and I am stuck on this excerpt on pg. 37 for a day now:

...This theory requires a loss function $L(Y, f(X))$ for penalizing errors in prediction, and by far the most common and convenient is squared error loss: $L(Y, f(X)) = (Y − f(X))$

This leads us to a criterion for choosing $f$, $$EPE(f) = E(Y − f(X))^2$$ $$= \int [y-f(x)]^2 Pr(dx, dy)$$

the expected (squared) prediction error. By conditioning on X, we can write EPE as $$EPE(f) = E_X E_{Y|X} ([Y - f(X)]^2 | X)$$

and we see that it suffices to minimize EPE pointwise: $$f(x) = argmin_cE_{Y|X} ([Y - c]^2 | X = x)$$

The solution is: $$f(x) = E(Y |X = x)$$

I have got most of it now but ultimately I do not understand how we arrived at solution i.e. how did we solve the $argmin(.)$ step to get the final regression function; is it supposed to be intuitive or is there a particular technique to solve it? I only just learnt about $argmin$ function and its definition doesn't offer much explanation as to what is going on here. It is doubly important because they later on change the loss function and same approach is required later on.

Thank you!

I'm not sure all that effort by Zhanxiong was necessary. Just expand $$E_{Y|X}([Y-c]^2) = E_{Y|X}Y^2 - 2cE_{Y|X}Y + c^2$$ and note that $c = E_{Y|X}Y$ minimizes the expression (derivative, square completion)...
• In general, $E(Y^2) \neq (E(Y))^2$. – Zhanxiong Dec 7 '14 at 0:43
• To show, it's a minimum, the second derivative of the expression with respect to $c$ must be positive. The second derivative here is just $2c$ which is equal to $E_{Y|X} Y$. Is this conditional expectation always positive? Averages can be negative too, right? – noir1993 Oct 4 at 15:10
This can be shown by a classical method used everywhere related to least squares estimation and conditional expectation. Let $f(x) = E(Y|X = x)$, then write: $$E_{Y|X}[(Y - c)^2|X = x] = E_{Y|X}[(Y - f(x) + f(x) - c)^2|X = x]$$ Expand the complete square and show that the cross product term is 0 as follows: $$E_{Y|X}[(Y - f(x))(f(x) - c)|X = x] = (f(x) - c)E_{Y|X}[Y - f(x)|X = x] = (f(x) - c)(f(x) - f(x)) = 0$$ where the first equality follows from $f(x) - c$ is a function of $x$ (technically, $\sigma(X)$-measurable) thus can be taken out from the conditional expectation. The second equality holds due to the linearity of expectation and our definition of $f(x)$. Therefore, $$E_{Y|X}[(Y - c)^2|X = x] = E_{Y|X}[(Y - f(x) + f(x) - c)^2|X = x] = E_{Y|X}[(Y - f(x))^2|X = x] + (f(x) - c)^2 \geq E_{Y|X}[(Y - f(x))^2|X = x]$$ And the equality can be attained by taking $c = f(x)$, which is the solution.