Significance versus the bias-variance trade off

Question

I'm taking an online course. In one of the lessons about methods to select explanatory variables, it is said that you can use the t-test or F-test to add/remove a single or group of terms to/from a model however, it is said that these tests are only concerned with the significance, and do not incorporate the bias-variance trade off. What does this mean?

Answer 1

Essentially, when choosing to include a particular variable in a model, we can consider the tradeoff between the amount of bias in our estimate, and the variance of our estimate.

Consider a case in which the true underlying model is defined as:

$y=\beta _0+\beta_1x_1+\beta_2x_2+u$

where $x_1$ and $x_2$ are correlated.

In this case, we compare two different models. The first using both $x_1$ and $x_2$ and the second using only $x_1$. We can define these models as:

1) $\hat{y}=\hat{\beta _0}+\hat{\beta _1}x_1+\hat{\beta _2}x_2$

2) $\widetilde{y}=\widetilde{\beta}_{0}+\widetilde{\beta}_{1}x_1$

The omission of $x_2$ in model 2 will result in a degree of bias in the estimate of $\beta_1$, equal to the slope coefficient from regressing $x_2$ on $x_1$ ($\delta_1$). Essentially:

$\widetilde{\beta}_{1}=\hat{\beta}_{1}+\hat{\beta}_{2}\widetilde{\delta}_{1}$

This is why multiple linear regression is preferred to performing multiple simple linear regressions, using individual correlated independent variables. However, we don't just consider the degree of bias, as including fewer independent variables (and estimated parameters) can decrease the variance in our parameter estimates. If we condition upon values of $x_1$ and $x_2$ in the sample we have:

$Var(\hat{\beta_1})-\sigma^2/[SST_1(1-R^2_1]$

In the above $SST_1$ is the variation in $x_1$ and $R^2_1$ is the R-squared regression $x_1$ on $x_2$. For the single independent variable case:

$Var(\widetilde{\beta}_{1})=\sigma^2/SST_1$.

Note that $Var(\widetilde{\beta}_{1})$ is always smaller than $Var(\hat{\beta}_{1})$

And a smaller variance in our parameter estimates, leads to greater precision in our estimates. So basically the trade-off can be summarised as:

"How much bias am I willing to accept in my estimates for increased precision?

The notation and general problems draws from Wooldridge (2006), and I would highly recommend reading Chapter 3 and 4 which goes into greater detail on the subject. General information on the role of significance testing is widely available, but these issues will not be addressed by simply performing a significance test.

References

Wooldridge, J. M. 2006. Introductory econometrics: a modern approach. Mason, OH, Thomson/South-Western.