Question about bias-variance tradeoff

Question

I'm trying to understand the bias-variance tradeoff, the relationship between the bias of the estimator and the bias of the model, and the relationship between the variance of the estimator and the variance of the model.

I came to these conclusions:

• We tend to overfit the data when we neglect the bias of the estimator, that is when we only aim to minimize the bias of the model neglecting the variance of the model (in other words we only aim to minimize the variance of the estimator without considering the bias of the estimator too)
• Vice versa, we tend to underfit the data when we neglect the variance of the estimator, that is when we only aim to minimize the variance of the model neglecting the bias of the model (in other words we only aim to minimize the bias of the estimator without considering the variance of the estimator too).

Are my conclusions correct?

Answer 1

Well, sort of. As stated, you ascribe intent to the scientist to minimize either bias or variance. In practice, you cannot explicitly observe the bias or the variance of your model (if you could, then you would know the true signal, in which case you wouldn't need a model). In general, you can only observe the error rate of your model on a specific data set, and you seek to estimate the out of sample error rate using various creative techniques.

Now you do know that, theoretically at least, this error rate can be decomposed into bias and variance terms, but you cannot directly observe this balance in any specific concrete situation. So I'd restate your observations slightly as:

• A model is underfit to the data when the bias term contributes the majority of out of sample error.
• A model is overfit to the data when the variance term contributes the majority of out of sample error.

In general, there is no real way to know for sure, as you can never truly observe the model bias. Nonetheless, there are various patterns of behavior that are indicative of being in one situation or another:

• Overfit models tend to have much worse goodness of fit performance on a testing dataset vs. a training data set.
• Underfit models tend to have the similar goodness of fit performance on a testing vs. training data set.

These are the patterns that are manifest in the famous plots of error rates by model complexity, this one is from The Elements of Statistical Learning:

Oftentimes these plots are overlaid with a bias and variance curve. I took this one from this nice exposition:

But, it is very important to realize that you never actually get to see these additional curves in any realistic situation.

Answer 2

Illustrating the Bias - Variance Tradeoff using a toy example

As @Matthew Drury points out, in realistic situations you don't get to see the last graph, but the following toy example may provide visual interpretation and intuition to those who find it helpful.

Dataset and assumptions

Consider the dataset which consists of i.i.d. samples from $Y$ a random variable defined as

• $Y = sin(\pi x - 0.5) + \epsilon$ where $\epsilon \sim Uniform(-0.5,0.5)$, or in other words
• $Y = f(x) + \epsilon$

Note that $x$ is not a random variable hence the variance of $Y$ is $Var(Y) = Var(\epsilon) = \frac{1}{12}$

We will be fitting a linear, polynomial regression model to this dataset of the form $ \hat f(x) = \beta_0 + \beta_1x + \beta_1 x^2 + ... + \beta_px^p$.

Fitting various polynomials models

Intuitively, you would expect a straight line curve to perform badly as the dataset is clearly non linear. Similarly, fitting a very high order polynomial might be excessive. This intuition is reflected in the graph below which shows the various models and their corresponding Mean Square Error for train and test data.

The above graph works for a single train / test split but how do we know whether it generalizes?

Estimating the expected train and test MSE

Here we have many options, but one approach is to randomly split the data between train / test - fit the model on the given split, and repeat this experiment many times. The resulting MSE can be plotted and the average is an estimate of the expected error.

It is interesting to see that the test MSE fluctuates wildly for different train / test splits of the data. But taking the average on a sufficiently large number of experiments gives us better confidence.

Note the gray dotted line that shows the variance of $Y$ computed at the beginning. It appears that on average the test MSE is never below this value

 Bias - Variance Decomposition

As explained here the MSE can be broken down into 3 main components:

$$E[ (Y - \hat f)^2 ] = \sigma^2_\epsilon + Bias^2[\hat f] + Var[\hat f]$$ $$E[ (Y - \hat f)^2 ] = \sigma^2_\epsilon + \left[ f - E[\hat f] \right]^2 + E\left[ \hat f - E[ \hat f] \right]^2$$

Where in our toy case:

• $f$ is known from the initial dataset
• $\sigma^2_\epsilon $ is known from the uniform distribution of $\epsilon$
• $E[\hat f]$ can be computed as above
• $\hat f$ corresponds to a lightly colored line
• $E\left[ \hat f - E[ \hat f] \right]^2$ can be estimated by taking the average

Giving the following relation

Note: the graph above uses the training data to fit the model and then calculates the MSE on train + test.