5
$\begingroup$

In Wikipedia, the "Bias–variance tradeoff" is mentioned in the context of prediction models where one can control the complexity of the model with some tuning parameters, and the more complex the model the more likely it is that the model would be less biased but with more variance.

However, there are other cases for Bias–variance tradeoff. For example, when estimating the error rate of a prediction model we are affected by which proportion of the observations we allocate for training vs testing. The more observations we have for training, the more close we will be to predict the error rate of a model which uses all the observations (less bias), but also the more variable would be our prediction of the error rate (higher variance).

Are there other classical/new cases which demonstrate some sort of a bias-variance tradeoff?

$\endgroup$
3
$\begingroup$

I recently wrote a (hopefully) accessible article to illustrate the problem to a non-technical audience in the context of time series forecasting: Kolassa (2016). "Sometimes it's better to be simple than correct", Foresight, 40:20-26.

I start by simulating 10,000 monthly time series of length 12 with a known but weak seasonality. The seasonality is easily visible when aggregating data:

total sales

However, it is invisible in disaggregate data, e.g., in the first five of the 10,000 series:

five time series

Now, assume that we fit two models to each separate series, a misspecified simple one using only the intercept,

$$y_t = \beta_0+\epsilon,$$

and a correct regression model regressing simulated sales on the (known and common) seasonal effects,

$$y_t = \beta_0+\beta_1 s_t+\epsilon.$$

Here are one year ahead forecasts for the first five series from the two models:

forecasts

We see that the seasonal shape is sometimes upside down, simply because of the noise in the simulated series.

Here are mean squared errors for both models per holdout month:

MSEs

We see that the misspecified simpler model always has lower errors than the correctly specified more complex one.

Finally, here is the connection to bias and variance - these are violin plots of the parameter estimates for both models across the 10,000 series (the dashed line indicates the true parameter values):

violin plots of parameter estimates

We notice that the correct model (of course) yields estimates that are unbiased, i.e., they are distributed around the true value. Conversely, the simpler misspecified model has biased parameter estimates: the intercept is on average too high, and the seasonal coefficient is biased low (because it doesn't occur in the model, so it's implicitly always zero, when the true value is one).

However, the key point is that the correct model's parameter estimates are far more variable, i.e., their violin plots are spread out a lot more than the corresponding violin plots in the misspecified simple model. And this variance in parameter estimates directly translates into higher forecasting MSEs (whereas the lower bias will reduce MSEs).

The bottom line is that when we are interested in forecasting or prediction, we shouldn't only care about the bias of our parameter estimates, but also about their variance. A larger model will (usually) have lower bias, but higher variance, and especially when we fit weak signals with little data, the higher variance may lead to larger forecasting errors. Shrinkage may help, by increasing bias, but reducing variance, hopefully in a way that total error is reduced.

$\endgroup$
1
  • $\begingroup$ Hmm, it seems you addressed the case that the OP was already aware of, but he is asking for other cases. Or am I too hasty to conclude this way? $\endgroup$ – Richard Hardy Jul 27 '16 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.