0
$\begingroup$

I'm looking at equation 7.9, section 7.3, page 223 in the book "The elements of statistical learning" in which the mean square error of the regression-estimator is decomposed into bias and variance terms. In my opinion the equations are very problematic as they are written. But let's see what I mean by "problematic" here.

We assume the relation $Y=f(X)+\epsilon$, for some "true" function $f$ and $\epsilon$ with some "true" fixed distribution. In the regression problem, we approximate the true $f$ by the regression function $\hat{f}$ which is derived by the std least squares method, based on the realisations of the random $X, Y$.

$\textbf{The problem}$: For an input $X=x_0$, the author defines (pointwise) a square error loss,

$Err(x_0) = \mathbb{E}[ (Y - \hat{f}(x_0) )^2 | X = x_0 ]$

The first discrepancy here (in my opinion) is that for a fixed vector $x_0$, both $\hat{f}(x_0)$ and $f(x_0)$ are both deterministic and the only stochastic term here is the error term (which in principle is unknown!) Then in the same equation (7.9), he defines bias as:

$\textbf{Bias} = \mathbb{E}[ \hat{f}(x_0)] - f(x_0) $. But $\hat{f}(x_0)$ is deterministic as $\hat{f}$ is an already calibrated function! Correct me if I'm wrong but I think I haven't lost my brain yet. Similarly, there is not such a thing as variance of the deterministic, $\hat{f}(x_0)$, $Var(\hat{f}(x_0))$.

At least for me, what is meaningful to define is the following prediction/generalisation error:

$Err = \mathbb{E}[ (Y - \hat{f}(X) )^2 ]$

with bias and variance (of the $\hat{f}(X)$ estimator of the real $f(X)$) defined as:

$\textbf{Bias} = \mathbb{E}[ \hat{f}(X) - f(X) ], \hspace{3mm} Var(\hat{f}(X))$.

Do I say something wrong here or the notation of the author is completely wrong in this equation?? By the way I have the best feelings for the author.

$\endgroup$

1 Answer 1

3
$\begingroup$

$\textbf{The problem}$: For an input $X=x_0$, the author defines (pointwise) a square error loss,

$Err(x_0) = \mathbb{E}[ (Y - \hat{f}(x_0) )^2 | X = x_0 ]$

The first discrepancy here (in my opinion) is that for a fixed vector $x_0$, both $\hat{f}(x_0)$ and $f(x_0)$ are both deterministic and the only stochastic term here is the error term (which in principle is unknown!)

Yes, but the formula does not take the expectation over $f(x_0)$, but over $Y$, which is stochastic. It could indeed be written as

$$Err(x_0) = \mathbb{E}[ (f(x_0)+\epsilon - \hat{f}(x_0) )^2 | X = x_0 ]$$

with a nonstochastic $f(x_0)$. However, $\hat{f}(x_0)$ is still stochastic:

Then in the same equation (7.9), he defines bias as:

$\textbf{Bias} = \mathbb{E}[ \hat{f}(x_0)] - f(x_0) $. But $\hat{f}(x_0)$ is deterministic as $\hat{f}$ is an already calibrated function!

$\hat{f}(x_0)$ is only calibrated conditional on the training data. The bias and variance of our model $\hat{f}$ are a challenge precisely because they depend on what we saw in our training data, which is where we may fall into overfitting.

Correct me if I'm wrong but I think I haven't lost my brain yet. Similarly, there is not such a thing as variance of the deterministic, $\hat{f}(x_0)$, $Var(\hat{f}(x_0))$.

See above: $\hat{f}$ is variable because of its dependency on the training data, so it makes sense to consider the variance of the value $\hat{f}(x_0)$ as a random variable that depends on the training data.

As an exercise, you could take input data $X$, simulate $Y$, estimate $\hat{f}$ in a correctly specified model, and apply $\hat{f}$ to $x_0$. Do this for fixed $X$ and $x_0$ but stochastic $Y$, and take a look at the variance of $\hat{f}(x_0)$. Or misspecify your model used in estimating $\hat{f}$, run through the same exercise, and study both the bias and the variance of $\hat{f}(x_0)$.

$\endgroup$
1
  • $\begingroup$ " it makes sense to consider the variance of the value 𝑓̂ (𝑥0) as a random variable that depends on the training data" - I see your point. From that perspective it makes sense. In general when we compute the generalisation error of a model we usually refer to a fixed model (here regression function) not the random-model calibrated from the random selection of the sample-X-y random vectors. Anyway, thank you for the reply. $\endgroup$ Apr 9, 2020 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.