# decomposition of MSE into Bias and Variance - Regression

I'm looking at equation 7.9, section 7.3, page 223 in the book "The elements of statistical learning" in which the mean square error of the regression-estimator is decomposed into bias and variance terms. In my opinion the equations are very problematic as they are written. But let's see what I mean by "problematic" here.

We assume the relation $$Y=f(X)+\epsilon$$, for some "true" function $$f$$ and $$\epsilon$$ with some "true" fixed distribution. In the regression problem, we approximate the true $$f$$ by the regression function $$\hat{f}$$ which is derived by the std least squares method, based on the realisations of the random $$X, Y$$.

$$\textbf{The problem}$$: For an input $$X=x_0$$, the author defines (pointwise) a square error loss,

$$Err(x_0) = \mathbb{E}[ (Y - \hat{f}(x_0) )^2 | X = x_0 ]$$

The first discrepancy here (in my opinion) is that for a fixed vector $$x_0$$, both $$\hat{f}(x_0)$$ and $$f(x_0)$$ are both deterministic and the only stochastic term here is the error term (which in principle is unknown!) Then in the same equation (7.9), he defines bias as:

$$\textbf{Bias} = \mathbb{E}[ \hat{f}(x_0)] - f(x_0)$$. But $$\hat{f}(x_0)$$ is deterministic as $$\hat{f}$$ is an already calibrated function! Correct me if I'm wrong but I think I haven't lost my brain yet. Similarly, there is not such a thing as variance of the deterministic, $$\hat{f}(x_0)$$, $$Var(\hat{f}(x_0))$$.

At least for me, what is meaningful to define is the following prediction/generalisation error:

$$Err = \mathbb{E}[ (Y - \hat{f}(X) )^2 ]$$

with bias and variance (of the $$\hat{f}(X)$$ estimator of the real $$f(X)$$) defined as:

$$\textbf{Bias} = \mathbb{E}[ \hat{f}(X) - f(X) ], \hspace{3mm} Var(\hat{f}(X))$$.

Do I say something wrong here or the notation of the author is completely wrong in this equation?? By the way I have the best feelings for the author.

$$\textbf{The problem}$$: For an input $$X=x_0$$, the author defines (pointwise) a square error loss,

$$Err(x_0) = \mathbb{E}[ (Y - \hat{f}(x_0) )^2 | X = x_0 ]$$

The first discrepancy here (in my opinion) is that for a fixed vector $$x_0$$, both $$\hat{f}(x_0)$$ and $$f(x_0)$$ are both deterministic and the only stochastic term here is the error term (which in principle is unknown!)

Yes, but the formula does not take the expectation over $$f(x_0)$$, but over $$Y$$, which is stochastic. It could indeed be written as

$$Err(x_0) = \mathbb{E}[ (f(x_0)+\epsilon - \hat{f}(x_0) )^2 | X = x_0 ]$$

with a nonstochastic $$f(x_0)$$. However, $$\hat{f}(x_0)$$ is still stochastic:

Then in the same equation (7.9), he defines bias as:

$$\textbf{Bias} = \mathbb{E}[ \hat{f}(x_0)] - f(x_0)$$. But $$\hat{f}(x_0)$$ is deterministic as $$\hat{f}$$ is an already calibrated function!

$$\hat{f}(x_0)$$ is only calibrated conditional on the training data. The bias and variance of our model $$\hat{f}$$ are a challenge precisely because they depend on what we saw in our training data, which is where we may fall into overfitting.

Correct me if I'm wrong but I think I haven't lost my brain yet. Similarly, there is not such a thing as variance of the deterministic, $$\hat{f}(x_0)$$, $$Var(\hat{f}(x_0))$$.

See above: $$\hat{f}$$ is variable because of its dependency on the training data, so it makes sense to consider the variance of the value $$\hat{f}(x_0)$$ as a random variable that depends on the training data.

As an exercise, you could take input data $$X$$, simulate $$Y$$, estimate $$\hat{f}$$ in a correctly specified model, and apply $$\hat{f}$$ to $$x_0$$. Do this for fixed $$X$$ and $$x_0$$ but stochastic $$Y$$, and take a look at the variance of $$\hat{f}(x_0)$$. Or misspecify your model used in estimating $$\hat{f}$$, run through the same exercise, and study both the bias and the variance of $$\hat{f}(x_0)$$.

• " it makes sense to consider the variance of the value 𝑓̂ (𝑥0) as a random variable that depends on the training data" - I see your point. From that perspective it makes sense. In general when we compute the generalisation error of a model we usually refer to a fixed model (here regression function) not the random-model calibrated from the random selection of the sample-X-y random vectors. Anyway, thank you for the reply. – noob-mathematician Apr 9 '20 at 14:33