Bias-Variance decomposition derivation

Question

My stats professor assigned this problem: "Show that the expected prediction error (EPE) for a squared error loss when $Y=f(X)+\varepsilon$ with estimator $\hat{f}(x)$ assuming $X=x$ is fixed and $\varepsilon~(0,\sigma^2)$ can be written as a combination of the bias and variance. In other words, show that

$$EPE(x) = E[(Y-\hat{f}(x))^2] = \sigma^2 + \text{Bias}^2 + \text{Var}(\hat{f}(x)).$$

I've come up with 3 ways to derive this relationship, but all of them depend on the assumption that $Y=f(X)+\varepsilon$ and $\hat{f}(X)$ are independent, or at least that their covariance is zero. For example, this allows me to use $\text{Var}[Y-\hat{f}(X)] = \text{Var}(Y)+\text{Var}(\hat{f}(X))$. This assumption makes intuitive sense to me because there is no necessary connection between $Y$ and the estimated value $\hat{f}(X)$ (the estimator could be a random number generator, after all). But I'm struggling to justify the independence assumption in a rigorous way. Can somebody nudge me toward understanding?

Answer 1

Here is a hint: consider $Y - \hat f = (Y - f) + (f - \hat f)$, and remember that $E(Y-f)=0$ and that $f$ is not random. Also, as @GeoMatt22 pointed out, you'll need $Cov(\varepsilon_0, \hat f) = 0$, which we get by virtue of iid errors.

(Basically I think you're probably making this more complicated than it needs to be, and it really just boils down to my hint)

Regarding whether or not $\hat f \perp Y$, generally our predictions are not just functions of $X$ but also of $Y$ so they can't be independent. In linear regression, for example, our fitted values $\hat Y = X(X^T X)^{-1}X^T Y$ so certainly it is not the case that $\hat Y \perp Y$ in general.

Update

I think the issue is that we've both been a little careless with what '$\varepsilon$' is. We observed data $(\bf y, \bf X)$ where in our data $y_i = f(x_i) + \varepsilon_i$, so that $\hat f$ is a function of $\bf y$, $\bf X$, and $\varepsilon_i$ for $i = 1, \dots, n$. We now observe a new point $(y_0, x_0)$ where we assume that $y_0 = f(x_0) + \varepsilon_0$. This is the key: this new point has its own error $\varepsilon_0$ that is independent of everything that went into $\hat f$ by the usual assumption of iid errors. So for $i = 1, \dots, n$ it definitely is not the case that $\varepsilon_i \perp \hat f$; but the error for a new point is indeed uncorrelated.