decomposition of MSE into Bias and Variance - Regression

Question

I'm looking at equation 7.9, section 7.3, page 223 in the book "The elements of statistical learning" in which the mean square error of the regression-estimator is decomposed into bias and variance terms. In my opinion the equations are very problematic as they are written. But let's see what I mean by "problematic" here.

We assume the relation $Y=f(X)+\epsilon$, for some "true" function $f$ and $\epsilon$ with some "true" fixed distribution. In the regression problem, we approximate the true $f$ by the regression function $\hat{f}$ which is derived by the std least squares method, based on the realisations of the random $X, Y$.

$\textbf{The problem}$: For an input $X=x_0$, the author defines (pointwise) a square error loss,

$Err(x_0) = \mathbb{E}[ (Y - \hat{f}(x_0) )^2 | X = x_0 ]$

The first discrepancy here (in my opinion) is that for a fixed vector $x_0$, both $\hat{f}(x_0)$ and $f(x_0)$ are both deterministic and the only stochastic term here is the error term (which in principle is unknown!) Then in the same equation (7.9), he defines bias as:

$\textbf{Bias} = \mathbb{E}[ \hat{f}(x_0)] - f(x_0) $. But $\hat{f}(x_0)$ is deterministic as $\hat{f}$ is an already calibrated function! Correct me if I'm wrong but I think I haven't lost my brain yet. Similarly, there is not such a thing as variance of the deterministic, $\hat{f}(x_0)$, $Var(\hat{f}(x_0))$.

At least for me, what is meaningful to define is the following prediction/generalisation error:

$Err = \mathbb{E}[ (Y - \hat{f}(X) )^2 ]$

with bias and variance (of the $\hat{f}(X)$ estimator of the real $f(X)$) defined as:

$\textbf{Bias} = \mathbb{E}[ \hat{f}(X) - f(X) ], \hspace{3mm} Var(\hat{f}(X))$.

Do I say something wrong here or the notation of the author is completely wrong in this equation?? By the way I have the best feelings for the author.

Answer 1

$\textbf{The problem}$: For an input $X=x_0$, the author defines (pointwise) a square error loss,

$Err(x_0) = \mathbb{E}[ (Y - \hat{f}(x_0) )^2 | X = x_0 ]$

The first discrepancy here (in my opinion) is that for a fixed vector $x_0$, both $\hat{f}(x_0)$ and $f(x_0)$ are both deterministic and the only stochastic term here is the error term (which in principle is unknown!)

Yes, but the formula does not take the expectation over $f(x_0)$, but over $Y$, which is stochastic. It could indeed be written as

$$Err(x_0) = \mathbb{E}[ (f(x_0)+\epsilon - \hat{f}(x_0) )^2 | X = x_0 ]$$

with a nonstochastic $f(x_0)$. However, $\hat{f}(x_0)$ is still stochastic:

Then in the same equation (7.9), he defines bias as:

$\textbf{Bias} = \mathbb{E}[ \hat{f}(x_0)] - f(x_0) $. But $\hat{f}(x_0)$ is deterministic as $\hat{f}$ is an already calibrated function!

$\hat{f}(x_0)$ is only calibrated conditional on the training data. The bias and variance of our model $\hat{f}$ are a challenge precisely because they depend on what we saw in our training data, which is where we may fall into overfitting.

Correct me if I'm wrong but I think I haven't lost my brain yet. Similarly, there is not such a thing as variance of the deterministic, $\hat{f}(x_0)$, $Var(\hat{f}(x_0))$.

See above: $\hat{f}$ is variable because of its dependency on the training data, so it makes sense to consider the variance of the value $\hat{f}(x_0)$ as a random variable that depends on the training data.

As an exercise, you could take input data $X$, simulate $Y$, estimate $\hat{f}$ in a correctly specified model, and apply $\hat{f}$ to $x_0$. Do this for fixed $X$ and $x_0$ but stochastic $Y$, and take a look at the variance of $\hat{f}(x_0)$. Or misspecify your model used in estimating $\hat{f}$, run through the same exercise, and study both the bias and the variance of $\hat{f}(x_0)$.