8
$\begingroup$

My stats professor assigned this problem: "Show that the expected prediction error (EPE) for a squared error loss when $Y=f(X)+\varepsilon$ with estimator $\hat{f}(x)$ assuming $X=x$ is fixed and $\varepsilon~(0,\sigma^2)$ can be written as a combination of the bias and variance. In other words, show that

$$EPE(x) = E[(Y-\hat{f}(x))^2] = \sigma^2 + \text{Bias}^2 + \text{Var}(\hat{f}(x)).$$

I've come up with 3 ways to derive this relationship, but all of them depend on the assumption that $Y=f(X)+\varepsilon$ and $\hat{f}(X)$ are independent, or at least that their covariance is zero. For example, this allows me to use $\text{Var}[Y-\hat{f}(X)] = \text{Var}(Y)+\text{Var}(\hat{f}(X))$. This assumption makes intuitive sense to me because there is no necessary connection between $Y$ and the estimated value $\hat{f}(X)$ (the estimator could be a random number generator, after all). But I'm struggling to justify the independence assumption in a rigorous way. Can somebody nudge me toward understanding?

$\endgroup$

1 Answer 1

5
$\begingroup$

Here is a hint: consider $Y - \hat f = (Y - f) + (f - \hat f)$, and remember that $E(Y-f)=0$ and that $f$ is not random. Also, as @GeoMatt22 pointed out, you'll need $Cov(\varepsilon_0, \hat f) = 0$, which we get by virtue of iid errors.

(Basically I think you're probably making this more complicated than it needs to be, and it really just boils down to my hint)

Regarding whether or not $\hat f \perp Y$, generally our predictions are not just functions of $X$ but also of $Y$ so they can't be independent. In linear regression, for example, our fitted values $\hat Y = X(X^T X)^{-1}X^T Y$ so certainly it is not the case that $\hat Y \perp Y$ in general.

Update

I think the issue is that we've both been a little careless with what '$\varepsilon$' is. We observed data $(\bf y, \bf X)$ where in our data $y_i = f(x_i) + \varepsilon_i$, so that $\hat f$ is a function of $\bf y$, $\bf X$, and $\varepsilon_i$ for $i = 1, \dots, n$. We now observe a new point $(y_0, x_0)$ where we assume that $y_0 = f(x_0) + \varepsilon_0$. This is the key: this new point has its own error $\varepsilon_0$ that is independent of everything that went into $\hat f$ by the usual assumption of iid errors. So for $i = 1, \dots, n$ it definitely is not the case that $\varepsilon_i \perp \hat f$; but the error for a new point is indeed uncorrelated.

$\endgroup$
6
  • $\begingroup$ I think the relevant assumption is that $\epsilon$ and $\hat{f}$ have zero covariance? $\endgroup$
    – GeoMatt22
    Sep 9, 2016 at 21:40
  • $\begingroup$ @GeoMatt22 good point. I've updated. $\endgroup$
    – jld
    Sep 9, 2016 at 21:50
  • $\begingroup$ Thanks. In one of my derivations I have $V(Y-\hat{f})=V(Y-f+f-\hat{f})=V(\varepsilon+f-\hat{f})=V(\varepsilon)+V(f)+V(\hat{f})=\sigma^2+0+V(\hat{f})$, which is what I want. But it seems that this argument rests on $\varepsilon$, $f$ and $\hat{f}$ having zero pairwise covariance, which gets back to my original problem. Am I missing something? $\endgroup$
    – gasbag_1
    Sep 9, 2016 at 21:51
  • $\begingroup$ $f$ is constant and you can show that a constant is uncorrelated with anything (proof: $Cov(X, a) = E(aX) - E(a)E(X) = a(EX-EX)=0$), so you really just need that $\varepsilon$ and $\hat f$ are uncorrelated $\endgroup$
    – jld
    Sep 9, 2016 at 21:53
  • $\begingroup$ Okay, that makes sense. So $\text{Cov}(\varepsilon,\hat{f}) = E(\varepsilon \hat{f})-E(\varepsilon)E(\hat{f}) = E(\varepsilon \hat{f})$. At this point I'm stuck. If $\hat{f}$ is a function of $Y$ and $Y=f(X)+\varepsilon$, then $\hat{f}$ and $\varepsilon$ are not independent. I would like to say that $E(\varepsilon \hat{f}) = \hat{f}E(\varepsilon) = 0$, but I don't see how to justify that move. Or must we simply assume that $\varepsilon$ and $\hat{f}$ are uncorrelated? $\endgroup$
    – gasbag_1
    Sep 9, 2016 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.