20
$\begingroup$

I am suffering from a blackout. I was presented the following picture to showcase the bias-variance tradeoff in the context of linear regression:

Polynomial model for data, simple and complex case

I can see that none of the two models is a good fit - the "simple" is not appreciating the complexity of the X-Y relation and the "complex" is just overfitting, basically learning the training data by heart. However I completely fail to see the bias and the variance in these two pictures. Could someone show this to me?

PS: The answer to Intuitive explanation of the bias-variance tradeoff? did not really help me, I would be glad if someone could provide a different approach based on the above picture.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
12
$\begingroup$

The bias variance trade-off is based on the breakdown of the mean square error:

$$MSE(\hat{y})=E[y-\hat{y}]^2=E[y-E[\hat{y}]]^2+E[\hat{y}-E[\hat{y}]]^2$$

One way to see the bias-variance trade of is what properties of the data set are used in the model fit. For the simple model, if we assume that OLS regression was used to fit the straight line, then only 4 numbers are used to fit the line:

  1. The sample covariance between x and y
  2. The sample variance of x
  3. The sample mean of x
  4. The sample mean of y

So, any graph which leads to the same 4 numbers above will lead to exactly the same fitted line (10 points, 100 points, 100000000 points). So in a sense it is insensitive to the particular sample observed. This means it will be "biased" because it effectively ignores part of the data. If that ignored part of the data happened to be important, then the predictions will be consistently in error. You will see this if you compare the fitted line using all data to the fitted lines obtained from removing one data point. They will tend to be quite stable.

Now the second model uses every scrap of data it can get, and fits the data as close as possible. Hence, the exact position of every data point matters, and so you can't shift the training data around without changing the fitted model like you can for OLS. Thus the model is very sensitive to the particular training set you have. The fitted model will be very different if you do the same drop-one data point plot.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The bias and variance of the model parameter estimate $\hat\theta$ or the predicted output value $\hat y$? Some people tell me that the terms bias and variance can only be used to describe the model parameter $\theta$, not the data $x,y$, is that right? $\endgroup$
    – avocado
    Dec 29, 2013 at 7:23
  • $\begingroup$ I don't think this is true, I think you are talking about prediction ($\hat {y} $) vs estimation ($\hat {\theta} $). Both have the notions of bias and variance - for example you have the "BLUE" for a regression parameter and "BLUP" for prediction a future data point. $\endgroup$
    – probabilityislogic
    Dec 29, 2013 at 20:46
  • $\begingroup$ For the parameter estimate $\hat\theta$, its bias is $bias(\hat\theta)=\theta-E[\hat\theta]$, but $\theta$ is unknown to us, right? Moreover, given the data set, we have no idea what the true model should look like, for example, the true model behind the data is $f(x)=a+bx+cx^2$, but we choose a linear regression model $h(x)=d+ex$ to fit the data, so here comes the paradox: the true parameters are $(a,b,c)$, which are the goal we should try to estimate, but we end up with the estimates of $(d,e)$, then compute or analyze the $bias(d)$ and $bias(e)$? $\endgroup$
    – avocado
    Dec 30, 2013 at 14:12
  • $\begingroup$ @loganecolss - this is not a paradox as the notion of bias only exists "locally" - that is, with respect to a given statistical model. The "paradox" exists for a person who: 1) knows the "true model", and 2) decides not to use it. That person is an idiot in my book. If you don't know the "true model" then there isn't a problem - unless you've found a good model and decided not to use it... $\endgroup$
    – probabilityislogic
    Dec 31, 2013 at 7:50
  • 1
    $\begingroup$ You have this fantasy of knowing a "true model" - it's not the right question to ask I think - its more a question of "does my current model have not enough or too many parameters?" - this does not depend on knowing what the "true model" is, and can be answered through standard model diagnostics. For example, why is your "true model" a function of variables that you have collected - and not a function like $ f (x, z_1, z_2,\dots, z_{K}) $ where you don't know 1) what the $ z_i$ values are, and 2) how many of them there are - ie you don't know $ K $. $\endgroup$
    – probabilityislogic
    Dec 31, 2013 at 22:32
5
$\begingroup$

To summarize with what I think I know in a non-mathematical manner:

  • bias - your prediction is going to be incorrect when you use the simple model and that will happen on any dataset you use the model on. Your prediction is expected to be wrong
  • variance - if you use the complex model, you will get very different prediction based on whichever dataset you are using

This page has a pretty good explanation with diagrams similar to what you posted. (I skipped the top part though, just read the part with diagrams) http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_bias_variance.htm (mouseover shows a different sample in case you didn't notice!)

Improve this answer
$\endgroup$
1
  • $\begingroup$ That's an interesting page and good illustrations, but I find them more confusing then helpful because (a) the "bias" and "variance" discussed in the context of regression do not appear to be the bias and variance as defined at the beginning of that page and (b) it's not at all clear that the statements being made (about how bias and variance change with number of parameters) are correct. $\endgroup$
    – whuber ♦
    Nov 29, 2011 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.