6
$\begingroup$

I was reading 'The Elements of Statistical Learning' and I am stuck on this excerpt on pg. 37 for a day now:

...This theory requires a loss function $L(Y, f(X))$ for penalizing errors in prediction, and by far the most common and convenient is squared error loss: $L(Y, f(X)) = (Y − f(X))$

This leads us to a criterion for choosing $f$, $$EPE(f) = E(Y − f(X))^2$$ $$ = \int [y-f(x)]^2 Pr(dx, dy)$$

the expected (squared) prediction error. By conditioning on X, we can write EPE as $$EPE(f) = E_X E_{Y|X} ([Y - f(X)]^2 | X)$$

and we see that it suffices to minimize EPE pointwise: $$f(x) = argmin_cE_{Y|X} ([Y - c]^2 | X = x)$$

The solution is: $$f(x) = E(Y |X = x)$$

I have got most of it now but ultimately I do not understand how we arrived at solution i.e. how did we solve the $argmin(.)$ step to get the final regression function; is it supposed to be intuitive or is there a particular technique to solve it? I only just learnt about $argmin$ function and its definition doesn't offer much explanation as to what is going on here. It is doubly important because they later on change the loss function and same approach is required later on.

Thank you!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

I'm not sure all that effort by Zhanxiong was necessary. Just expand $$E_{Y|X}([Y-c]^2) = E_{Y|X}Y^2 - 2cE_{Y|X}Y + c^2$$ and note that $c = E_{Y|X}Y$ minimizes the expression (derivative, square completion)...

Improve this answer
$\endgroup$
6
  • $\begingroup$ In general, $E(Y^2) \neq (E(Y))^2$. $\endgroup$
    – Zhanxiong
    Dec 7, 2014 at 0:43
  • $\begingroup$ Thank you, but how is this related to my answer? $\endgroup$
    – psarka
    Dec 7, 2014 at 0:54
  • $\begingroup$ Oh, I see, your answer is has no problem. sorry. $\endgroup$
    – Zhanxiong
    Dec 7, 2014 at 1:05
  • $\begingroup$ Removing the conditional expression made it clear. Thank you! $\endgroup$
    – Anurag Kalia
    Dec 7, 2014 at 1:19
  • $\begingroup$ To show, it's a minimum, the second derivative of the expression with respect to $c$ must be positive. The second derivative here is just $2c$ which is equal to $E_{Y|X} Y$. Is this conditional expectation always positive? Averages can be negative too, right? $\endgroup$
    – noir1993
    Oct 4, 2020 at 15:10
4
$\begingroup$

This can be shown by a classical method used everywhere related to least squares estimation and conditional expectation. Let $f(x) = E(Y|X = x)$, then write: $$E_{Y|X}[(Y - c)^2|X = x] = E_{Y|X}[(Y - f(x) + f(x) - c)^2|X = x]$$ Expand the complete square and show that the cross product term is 0 as follows: $$E_{Y|X}[(Y - f(x))(f(x) - c)|X = x] = (f(x) - c)E_{Y|X}[Y - f(x)|X = x] = (f(x) - c)(f(x) - f(x)) = 0$$ where the first equality follows from $f(x) - c$ is a function of $x$ (technically, $\sigma(X)$-measurable) thus can be taken out from the conditional expectation. The second equality holds due to the linearity of expectation and our definition of $f(x)$. Therefore, $$E_{Y|X}[(Y - c)^2|X = x] = E_{Y|X}[(Y - f(x) + f(x) - c)^2|X = x] = E_{Y|X}[(Y - f(x))^2|X = x] + (f(x) - c)^2 \geq E_{Y|X}[(Y - f(x))^2|X = x]$$ And the equality can be attained by taking $c = f(x)$, which is the solution.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Intuitively and geometrically, f(x), as conditional expectation, is the projection of Y onto the linear space spanned by X (predictor variables) such that the distance between Y and any vector belongs to X-space is minimized. Hence possesses the optimal property mentioned in your problem. $\endgroup$
    – Zhanxiong
    Dec 7, 2014 at 0:13
  • $\begingroup$ Doesn't this approach mean I should guess the function beforehand? It is fascinating though, thank you! $\endgroup$
    – Anurag Kalia
    Dec 7, 2014 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.