I am looking for an intuitive explanation of the bias-variance tradeoff, both in general and specifically in the context of linear regression.
Imagine some 2D data--let's say height versus weight for students at a high school--plotted on a pair of axes.
Now suppose you fit a straight line through it. This line, which of course represents a set of predicted values, has zero statistical variance. But the bias is (probably) high--i.e., it doesn't fit the data very well.
Next, suppose you model the data with a high-degree polynomial spline. You're not satisfied with the fit, so you increase the polynomial degree until the fit improves (and it will, to arbitrary precision, in fact). Now you have a situation with bias that tends to zero, but the variance is very high.
Note that the bias-variance trade-off doesn't describe a proportional relationship--i.e., if you plot bias versus variance you won't necessarily see a straight line through the origin with slope -1. In the polynomial spline example above, reducing the degree almost certainly increases the variance much less than it decreases the bias.
The bias-variance tradeoff is also embedded in the sum-of-squares error function. Below, I have rewritten (but not altered) the usual form of this equation to emphasize this:
On the right-hand side, there are three terms: the first of these is just the irreducible error (the variance in the data itself); this is beyond our control so ignore it. The second term is the square of the bias; and the third is the variance. It's easy to see that as one goes up the other goes down--they can't both vary together in the same direction. Put another way, you can think of least-squares regression as (implicitly) finding the optimal combination of bias and variance from among candidate models.
Let's say you are considering catastrophic health insurance, and there is a 1% probability of getting sick which would cost 1 million dollars. The expected cost of getting sick is thus 10,000 dollars. The insurance company, wanting to make a profit, will charge you 15,000 for the policy.
Buying the policy gives an expected cost to you of 15,000, which has a variance of 0 but can be thought of as biased since it is 5,000 more than the real expected cost of getting sick.
Not buying the policy gives an expected cost of 10,000, which is unbiased since it is equal to the true expected cost of getting sick, but has a very high variance. The tradeoff here is between an approach that is consistently wrong but never by much and an approach that is correct on average but is more variable.
I highly recommend having a look at Caltech ML course by Yaser Abu-Mostafa, Lecture 8 (Bias-Variance Tradeoff) . Here are the outlines:
Say you are trying to learn the sine function:
Our training set consists of only 2 data points.
Let's try to do it with two models, $h_0(x)=b$ and $h_1(x)=ax+b$:
For $h_0(x)=b$, when we try with many different training sets (i.e. we repeatedly select 2 data points and perform the learning on them), we obtain (left graph represents all the learnt models, right graph represent their mean g and their variance (grey area)):
For $h_1(x)=ax+b$, when we try with many different training sets, we obtain: if If we compare the learnt model with $h_0$ and $h_1$, we can see that $h_0$ yields more simple models than $h_1$, hence a lower variance when we consider all the models learnt with $h_0$, but the best model g (in red on the graph) learnt with $h_1$ is better than the best model learnt g with $h_0$, hence a lower bias with $h_1$: