Where does linear regression fit into the bias-variance tradeoff?

Question

In ISL, the concept of the bias-variance tradeoff is presented with the rule of thumb that simple models will have high bias and that complex models will have high variance.

Given this idea, I would think that linear regression would have a high bias, being a simple model with a huge assumption built in (that the data is linear); however, I am also aware that OLS is an unbiased estimator.

How do I reconcile these two facts?

Answer 1

OLS is an unbiased estimator assuming the model is true, which is to say,

1. Effects are exactly linear
2. All variables with non-zero effects are included
3. All interactions are included
4. no non-linear effects

and other small model inadequacies. See my answer at Why do irrelevant regressors become statistically significant in large samples?. The bias-variance decomposition is (from section 7.3 of ESL, second edition) $$\DeclareMathOperator{\E}{\mathbb{E}}\begin{align} \text{Err}(x_0)&=\E[(Y-\hat{f}(x_0)))^2\mid X=x_0] \\ &=\sigma^2_\epsilon + [\E \hat{f}(x_0)-f(x_0)]^2 + \E [\hat{f}(x_0)-\E\hat{f}(x_0)]^2 \\ &= \sigma^2_ \epsilon + \text{Bias}^2(\hat{f}(x_0)) + \text{Var}(\hat{f}(x_0))\\ &= \text{Irreducible Error}+\text{Bias}^2 + \text{Variance}. \end{align} $$ If your model is correct, the $\text{Bias}^2$ term will be zero, but if the model is approximate, it will not.

Answer 2

Linear regression is a general term. When used, $y=ax+b+\epsilon$ is what comes to mind first, however $y=ax^2+bx+c+\epsilon$ is also linear regression, i.e. $x_2=x, x_1=x^2$ and $y=ax_1+bx_2+c+\epsilon$. It's just we use polynomial features. The data (target) can be of parabolic nature but it can still be estimated via linear regression if you use polynomial features. High bias occurs when you use overly simplistic model compared to the data; not specifically when you use the model $y=ax+b+\epsilon$.

Unbiased estimator is a slightly different concept. If an estimator, say $\hat{\theta}$ for a variable $\theta$ is unbiased, then we have $E[\hat{\theta}]=\theta$. A very simple unbiased estimator is the mean; it is also unbiased since if $\hat{\theta}=\mu$ then the expected value of it will equal to the mean: $E[\hat{\theta}]=E[\mu]=\mu=E[\theta]$. Therefore, let alone OLS, using just the mean is an unbiased estimation technique. So, having an unbiased estimator doesn't mean that your estimator fits well to your data.