I recently wrote a (hopefully) accessible article to illustrate the problem to a non-technical audience in the context of time series forecasting: Kolassa (2016). "Sometimes it's better to be simple than correct", Foresight, 40:20-26.
I start by simulating 10,000 monthly time series of length 12 with a known but weak seasonality. The seasonality is easily visible when aggregating data:
However, it is invisible in disaggregate data, e.g., in the first five of the 10,000 series:
Now, assume that we fit two models to each separate series, a misspecified simple one using only the intercept,
$$y_t = \beta_0+\epsilon,$$
and a correct regression model regressing simulated sales on the (known and common) seasonal effects,
$$y_t = \beta_0+\beta_1 s_t+\epsilon.$$
Here are one year ahead forecasts for the first five series from the two models:
We see that the seasonal shape is sometimes upside down, simply because of the noise in the simulated series.
Here are mean squared errors for both models per holdout month:
We see that the misspecified simpler model always has lower errors than the correctly specified more complex one.
Finally, here is the connection to bias and variance - these are violin plots of the parameter estimates for both models across the 10,000 series (the dashed line indicates the true parameter values):
We notice that the correct model (of course) yields estimates that are unbiased, i.e., they are distributed around the true value. Conversely, the simpler misspecified model has biased parameter estimates: the intercept is on average too high, and the seasonal coefficient is biased low (because it doesn't occur in the model, so it's implicitly always zero, when the true value is one).
However, the key point is that the correct model's parameter estimates are far more variable, i.e., their violin plots are spread out a lot more than the corresponding violin plots in the misspecified simple model. And this variance in parameter estimates directly translates into higher forecasting MSEs (whereas the lower bias will reduce MSEs).
The bottom line is that when we are interested in forecasting or prediction, we shouldn't only care about the bias of our parameter estimates, but also about their variance. A larger model will (usually) have lower bias, but higher variance, and especially when we fit weak signals with little data, the higher variance may lead to larger forecasting errors. Shrinkage may help, by increasing bias, but reducing variance, hopefully in a way that total error is reduced.