Low Bias in an overfitted model

Question

I have a question about the bias-variance tradeoff in machine learning concerning the implications of overfitting:

Assuming $y = f(x)$ + some noise, the error of our model for any input $(x,y)$ is given by: \begin{eqnarray} E [y-\hat{f}(x)] &=& (E[f(x)-\hat{f}(x)]^2 + E[(\hat{f}(x) - E[\hat{f}(x)])^2] \\ &=& \text{Bias}(\hat f(x),f(x))^2 + \text{Var}(\hat f(x)) \\ \end{eqnarray}

I understand that overfitting will lead to a high variance in our model, regarding random training input.

I also understand that we will have small bias terms for all $(x_t, y_t)$-pairs, that we used to train our model. But I don't see at all why an overfitted model will lead to a small bias in general, for example for some validation pair $(x_v, y_v)$. Actually the behaviour of an overfitted model, outside of the training data, seems completly unreliable to me.

Is the bias, that we are talking about in the "bias-variance tradeoff" only aiming at the bias for the given training data predictions?

I would understand that, but what I wonder about than is that regarding the variance only for a given training set doesn't seem to make any sense, since we are actually talking about the variance in our model for some random training input, that we used to generate our estimator, right?

Answer 1

1. The expectation is with respect to draws of (y, x). So, when we are inside the expectation brackets, we are dealing with a given draw of (y, x).
2. $\hat{f}$, the estimated model, is a function of both y and x. Hence, it varies with different draws of the sample. $E[\hat{f}]$ is the average prediction of y using models estimated over various sample draws.
3. Thus, the bias term is a measure of "average" (over all the sample draws) of in-sample squared error (as we find the fitting error within the expectation ie. for a given sample).
4. The variance on the other hand is the average of the difference between (a) model estimate from this sample and (b) the average of estimates of models fit in all sample draws. This is the only place where we have a component of out-of-sample testing (through (b)). Thus, this is a measure of how much our estimates vary across samples.