complex models have low bias and high variance I have trouble understanding the trade-off between bias and variance. I can comprehend that complex models are better able to approach the "true distribution". Therefore, they have low bias. But why do they necessarily have high variance?
 A: Remember that we talk of variance in terms of parameter estimates across samples. That is, if we sample several different training sets and fit our model to each of those separately, what is the variance in the resulting parameter estimates?
A more complex model is much better able to fit the training data. The problem is that this can come in the form of oversensitivity. Instead of identifying the essential elements, you can overfit to noise in the data. The noise from sample to sample is different, so your variance is high. By contrast, a much simpler model lacks the capacity to do that.
I think the quintessential example is of fitting a polynomial to points sampled from a true curve. As you increase the order of your polynomial, you can certainly include all of the points—but the resulting polynomials will be vastly different depending on which points were sampled. By contrast, a low-order polynomial like a line or parabola may lack the capacity to pass through every point (high bias), but from sample to sample the parameter estimates will be more consistent (low variance).
A: +1 to Arya's answer.
Here is a little example to illustrate his point. Suppose we have some underlying relationship that we want to model. We sample a few points from it and fit models of varying complexity. What we are interested in is the variability of these models as we re-run the entire experiment (sampling and fitting) multiple times.
Below is a simulation. We run our experiment four times, yielding four panels. The true relationship is the dashed black line; it happens to be quadratic. In each experiment, we sample 20 data points, and in each case, we fit a model with a quadratic relationship (fitted values given by the red line), and a second model with a fifth-order relationship (the green line).

The key observation is that the green lines vary much more between experiments than the red lines. The red lines can still be off, e.g., they may be concave instead of convex - but at least they don't vary quite as much.
Of course, in any given experiment, you will see only one of these panels. And the problem here is that which one of the panels you get is down to chance - specifically, to which points you happened to sample. So if you run too complicated a model, your results will be a single draw from a highly variable distribution.
R code:
xx <- seq(0,1,by=.01); ylim <- c(-.5,1) # for plotting
true_relationship <- function(xx) xx^2-2/3*xx
ss <- 0.5
n_sample <- 20
degrees <- c(2,5)

par(mfrow=c(2,2),lwd=2,mai=c(.2,.2,.2,.2))
for ( ii in 1:4 ) {
    set.seed(ii)    # for reproducibility
    x_sample <- runif(n_sample)
    y_sample <- true_relationship(x_sample)+rnorm(n_sample,0,ss)
    plot(xx,true_relationship(xx),type="l",ylim=ylim,
        xlab="",ylab="",xaxt="n",yaxt="n",lty=2)
    points(x_sample,y_sample,pch=19)
    for ( jj in seq_along(degrees) ) {
        model <- lm(y_sample~poly(x_sample,degrees[jj]))
        lines(xx,predict(model,newdata=data.frame(x_sample=xx)),col=jj+1)
    }
}

