17
$\begingroup$

I am suffering from a blackout. I was presented the following picture to showcase the bias-variance tradeoff in the context of linear regression:

Polynomial model for data, simple and complex case

I can see that none of the two models is a good fit - the "simple" is not appreciating the complexity of the X-Y relation and the "complex" is just overfitting, basically learning the training data by heart. However I completely fail to see the bias and the variance in these two pictures. Could someone show this to me?

PS: The answer to Intuitive explanation of the bias-variance tradeoff? did not really help me, I would be glad if someone could provide a different approach based on the above picture.

$\endgroup$
11
$\begingroup$

The bias variance trade-off is based on the breakdown of the mean square error:

$$MSE(\hat{y})=E[y-\hat{y}]^2=E[y-E[\hat{y}]]^2+E[\hat{y}-E[\hat{y}]]^2$$

One way to see the bias-variance trade of is what properties of the data set are used in the model fit. For the simple model, if we assume that OLS regression was used to fit the straight line, then only 4 numbers are used to fit the line:

  1. The sample covariance between x and y
  2. The sample variance of x
  3. The sample mean of x
  4. The sample mean of y

So, any graph which leads to the same 4 numbers above will lead to exactly the same fitted line (10 points, 100 points, 100000000 points). So in a sense it is insensitive to the particular sample observed. This means it will be "biased" because it effectively ignores part of the data. If that ignored part of the data happened to be important, then the predictions will be consistently in error. You will see this if you compare the fitted line using all data to the fitted lines obtained from removing one data point. They will tend to be quite stable.

Now the second model uses every scrap of data it can get, and fits the data as close as possible. Hence, the exact position of every data point matters, and so you can't shift the training data around without changing the fitted model like you can for OLS. Thus the model is very sensitive to the particular training set you have. The fitted model will be very different if you do the same drop-one data point plot.

$\endgroup$
  • $\begingroup$ The bias and variance of the model parameter estimate $\hat\theta$ or the predicted output value $\hat y$? Some people tell me that the terms bias and variance can only be used to describe the model parameter $\theta$, not the data $x,y$, is that right? $\endgroup$ – avocado Dec 29 '13 at 7:23
  • $\begingroup$ I don't think this is true, I think you are talking about prediction ($\hat {y} $) vs estimation ($\hat {\theta} $). Both have the notions of bias and variance - for example you have the "BLUE" for a regression parameter and "BLUP" for prediction a future data point. $\endgroup$ – probabilityislogic Dec 29 '13 at 20:46
  • $\begingroup$ For the parameter estimate $\hat\theta$, its bias is $bias(\hat\theta)=\theta-E[\hat\theta]$, but $\theta$ is unknown to us, right? Moreover, given the data set, we have no idea what the true model should look like, for example, the true model behind the data is $f(x)=a+bx+cx^2$, but we choose a linear regression model $h(x)=d+ex$ to fit the data, so here comes the paradox: the true parameters are $(a,b,c)$, which are the goal we should try to estimate, but we end up with the estimates of $(d,e)$, then compute or analyze the $bias(d)$ and $bias(e)$? $\endgroup$ – avocado Dec 30 '13 at 14:12
  • $\begingroup$ @loganecolss - this is not a paradox as the notion of bias only exists "locally" - that is, with respect to a given statistical model. The "paradox" exists for a person who: 1) knows the "true model", and 2) decides not to use it. That person is an idiot in my book. If you don't know the "true model" then there isn't a problem - unless you've found a good model and decided not to use it... $\endgroup$ – probabilityislogic Dec 31 '13 at 7:50
  • $\begingroup$ Suppose I don't know the true model is $f(x)$, I pick $h(x)$ to fit the sample, and $\hat d,\hat e$ are the parameter estimates for $h(x)$, then how to measure the bias? Is it $bias(\hat d)=E[\hat d]-d_{true}$, where $d_{true}$ is the true parameter? $\endgroup$ – avocado Dec 31 '13 at 14:21
5
$\begingroup$

To summarize with what I think I know in a non-mathematical manner:

  • bias - your prediction is going to be incorrect when you use the simple model and that will happen on any dataset you use the model on. Your prediction is expected to be wrong
  • variance - if you use the complex model, you will get very different prediction based on whichever dataset you are using

This page has a pretty good explanation with diagrams similar to what you posted. (I skipped the top part though, just read the part with diagrams) http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_bias_variance.htm (mouseover shows a different sample in case you didn't notice!)

$\endgroup$
  • $\begingroup$ That's an interesting page and good illustrations, but I find them more confusing then helpful because (a) the "bias" and "variance" discussed in the context of regression do not appear to be the bias and variance as defined at the beginning of that page and (b) it's not at all clear that the statements being made (about how bias and variance change with number of parameters) are correct. $\endgroup$ – whuber Nov 29 '11 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.