In The Elements of Statistical Learning, at page 18 the authors explain that, in order to minimize the EPE (Expected Prediction Error defined as the mean of the loss function: $\text{EPE}(f) = \mathbb{E}[L(f)]$):

$$ \text{EPE}(f) = \mathbb{E}[(Y-f(X))^2] $$

Which is based on Loss function $L_2=(Y-f(X))^2$ (quadratic error), we need to have:

$$ f(X) = \mathbb{E}[Y|X=x] $$

Note The authors are using, in last equation, the model $f$ as solution. not the estimate $\hat{f}$!

However, later on, the authors consider another possible definition of the EPE based on Loss Function: $L_1 = |Y-f(X)|$ (absolute error):

$$ \text{EPE}(f) = \mathbb{E}[|Y-f(X)|] $$

And that to minimize it, we need to have (pg. 20):

$$ \hat{f}(X) = \text{median}(Y|X=x) $$

Why isn't this time the equation referring to the original model $f$ but its estimate $\hat{f}$?

It is my understanding that the minimizer of the EPE is a particular value of the model $f$ we choose to assume the data come from.

$L_2$ EPE

In the first set of equations, we understand that the $L_2$ EPE is minimized by using a model that is the conditional mean. However the model is always unknown, so we estimate it. Therefore, our objective becomes finding an estimation process that tries to get as close as possible to the conditional mean which represents the best possible model ever (an ideal solution we try to reach)!

$L_1$ EPE

Shouldn't this approach be applied also with the $L_1$ EPE?

  • Why do the authors say that the estimate $\hat{f}$ is the solution rather than saying that the model $f$ is the solution which minimizes the $L_1$ EPE?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.