# complex models have low bias and high variance

I have trouble understanding the trade-off between bias and variance. I can comprehend that complex models are better able to approach the "true distribution". Therefore, they have low bias. But why do they necessarily have high variance?

Remember that we talk of variance in terms of parameter estimates across samples. That is, if we sample several different training sets and fit our model to each of those separately, what is the variance in the resulting parameter estimates?

A more complex model is much better able to fit the training data. The problem is that this can come in the form of oversensitivity. Instead of identifying the essential elements, you can overfit to noise in the data. The noise from sample to sample is different, so your variance is high. By contrast, a much simpler model lacks the capacity to do that.

I think the quintessential example is of fitting a polynomial to points sampled from a true curve. As you increase the order of your polynomial, you can certainly include all of the points—but the resulting polynomials will be vastly different depending on which points were sampled. By contrast, a low-order polynomial like a line or parabola may lack the capacity to pass through every point (high bias), but from sample to sample the parameter estimates will be more consistent (low variance).

Here is a little example to illustrate his point. Suppose we have some underlying relationship that we want to model. We sample a few points from it and fit models of varying complexity. What we are interested in is the variability of these models as we re-run the entire experiment (sampling and fitting) multiple times.

Below is a simulation. We run our experiment four times, yielding four panels. The true relationship is the dashed black line; it happens to be quadratic. In each experiment, we sample 20 data points, and in each case, we fit a model with a quadratic relationship (fitted values given by the red line), and a second model with a fifth-order relationship (the green line).

The key observation is that the green lines vary much more between experiments than the red lines. The red lines can still be off, e.g., they may be concave instead of convex - but at least they don't vary quite as much.

Of course, in any given experiment, you will see only one of these panels. And the problem here is that which one of the panels you get is down to chance - specifically, to which points you happened to sample. So if you run too complicated a model, your results will be a single draw from a highly variable distribution.

R code:

xx <- seq(0,1,by=.01); ylim <- c(-.5,1) # for plotting
true_relationship <- function(xx) xx^2-2/3*xx
ss <- 0.5
n_sample <- 20
degrees <- c(2,5)

par(mfrow=c(2,2),lwd=2,mai=c(.2,.2,.2,.2))
for ( ii in 1:4 ) {
set.seed(ii)    # for reproducibility
x_sample <- runif(n_sample)
y_sample <- true_relationship(x_sample)+rnorm(n_sample,0,ss)
plot(xx,true_relationship(xx),type="l",ylim=ylim,
xlab="",ylab="",xaxt="n",yaxt="n",lty=2)
points(x_sample,y_sample,pch=19)
for ( jj in seq_along(degrees) ) {
model <- lm(y_sample~poly(x_sample,degrees[jj]))
lines(xx,predict(model,newdata=data.frame(x_sample=xx)),col=jj+1)
}
}

• Small thing: his point. Thanks for creating the plots—exactly the sort of thing I was describing. :) – Arya McCarthy Feb 26 at 13:28
• @AryaMcCarthy: sorry, I'll need to update my priors. Will do so, also my post. – Stephan Kolassa Feb 26 at 14:10
• Thank you very much for the plots and your explanation. I have another question. If we increase our sample size, will the variance decrease? How about bias? Because intuitively, if we have more data, models of higher complexity will be better able to fit the underlying distribution, so they won't differ very much from each other. On the other hand, no matter how many sample points we have, models of low complexity won't be able to mimic the true distribution. So the bias remains high. Am I correct? – Gracie Feb 27 at 3:15
• Yes indeed: increasing the number of data points will reduce both variance and bias of the estimated model (up to some minimal values that are determined by the residual variance of the process). So in principle, we could use highly complex models and just collect enough data. The problem is that the variance only goes down slowly as the sample size increases, so "enough" data may be a huge amount, more than is feasible. Usually, it's better to use some kind of regularization, rather than collect an infinite amount of data. – Stephan Kolassa Feb 27 at 7:10