Quality of a model and the bias-variance tradeoff

Take linear regression as the example, given one specific data set $D_1=\{(x_1,y_1),...(x_n,y_n)\}$, we could train a model with one specific parameter estimate $\hat\theta_1$, if we do the training on a new data set $D_2$, we will have new estimate $\hat\theta_2$, and so on. For input data $x$, I could predict the target value $\hat y$.

My problems are,

1. I thought the term bias and variance are only used when talking about the parameter estimate $\hat\theta$ of model, not used to describe the random variable of the predicted data, which are $\hat y$, right? So we have $bias(\hat\theta),var(\hat\theta)$, but don't have $bias(\hat y)$ or $var(\hat y)$. If I'm wrong, then what are bias and variance here?

2. To judge the quality of the one specific model with parameter estimate $\hat\theta_1$, I check the mean squared error over all the training data points of $D_1$, $$MSE_1=\frac{1}{n}\sum_i(y_i-\hat y_i)^2$$, but for different training data sets $D_i$, we have different $\hat\theta_i$, then different $MSE_i$, so how do I determine what $\hat\theta$ should be? The one with the smallest $MSE$?

3. As I read the bias-variance tradeoff, it's said that the expectation of the $MSE$ is of special interest, why should we pay special attention to the $E[MSE]$? I mean I could also pay attention to $E[\hat y_i]$ for each data point. Moreover, $MSE$ is calculated on the observed target value $y_i$ and its prediction $\hat y_i$, so I thought it has to with the $bias(\hat y_i)$ and $var(\hat y_i)$, but not the bias or variance of $\hat\theta$?

I totally got lost when trying to figure out what the terms bias and variance are aimed at. I hope anyone of you can help me out.

• (1) is wikipedia the source of your information? you might want to find another resource (2) the wikipedia article seems to define bias/variance quite specifically, which you've chosen to ignore in 1 & 2 above. – charles Dec 26 '13 at 20:41
• It is possible to speak about the potential bias & variance of the process / algorithm used to generate $\hat y$, but it's more complicated, as it is a function of the bias & variance of $\hat\beta_0$ & $\hat\beta_1$. I suggest you focus on understanding these concepts as applied to parameters first, where it's somewhat easier. On another note, the phrase "the random variable of the predicted data" makes no sense. – gung Dec 26 '13 at 22:26
• @gung, as applied to parameter estimate $\hat\theta$, different training sets $D_i$ will give different estimate $\hat\theta_i$, but according the $MSE(\hat\theta)=\frac{1}{n}\sum_iE[(\hat\theta_i-\theta_{truth})^2]$, I can measure how accurate the chosen model (choice between linear or nonlinear) is, right? And furthermore, $MSE(\hat\theta)$ can be decomposed into the sum of $bias^2(\hat\theta)+var(\hat\theta)$, right? – avocado Dec 27 '13 at 0:49
• @gung, that's why we have to strike a balance between $bias(\hat\theta)$ and $var(\hat\theta)$ to get a good model, isn't it? And since we don't know $\theta_{truth}$, so we can't know $bias(\hat\theta)$, right? – avocado Dec 27 '13 at 0:53