In ISL, the concept of the bias-variance tradeoff is presented with the rule of thumb that simple models will have high bias and that complex models will have high variance.

Given this idea, I would think that linear regression would have a high bias, being a simple model with a huge assumption built in (that the data is linear); however, I am also aware that OLS is an unbiased estimator.

How do I reconcile these two facts?

  • $\begingroup$ (though adding non linear transformations of your variables eg polynomial regression) is still linear (in parameters) regression $\endgroup$
    – seanv507
    Mar 12 '19 at 11:56

OLS is an unbiased estimator assuming the model is true, which is to say,

  1. Effects are exactly linear
  2. All variables with non-zero effects are included
  3. All interactions are included
  4. no non-linear effects

and other small model inadequacies. See my answer at Why do irrelevant regressors become statistically significant in large samples?. The bias-variance decomposition is (from section 7.3 of ESL, second edition) $$\DeclareMathOperator{\E}{\mathbb{E}}\begin{align} \text{Err}(x_0)&=\E[(Y-\hat{f}(x_0)))^2\mid X=x_0] \\ &=\sigma^2_\epsilon + [\E \hat{f}(x_0)-f(x_0)]^2 + \E [\hat{f}(x_0)-\E\hat{f}(x_0)]^2 \\ &= \sigma^2_ \epsilon + \text{Bias}^2(\hat{f}(x_0)) + \text{Var}(\hat{f}(x_0))\\ &= \text{Irreducible Error}+\text{Bias}^2 + \text{Variance}. \end{align} $$ If your model is correct, the $\text{Bias}^2$ term will be zero, but if the model is approximate, it will not.

  • $\begingroup$ I think the question refers to situations where the real phenomenon we are trying to model is not linear, but we use linear regression as an aproximation $\endgroup$
    – David
    Mar 12 '19 at 11:19
  • $\begingroup$ Yes, and that is what I am trying to say here! In some cases the bias^2 term can be further subdivided in model bias and estimation bias, when some nonlinear model is approximated by a linear model and OLS is used, the estimation boas will be zero, but nor the model bias. $\endgroup$ Mar 12 '19 at 11:22
  • 1
    $\begingroup$ Awesome thank you so much! $\endgroup$ Mar 12 '19 at 16:04

Linear regression is a general term. When used, $y=ax+b+\epsilon$ is what comes to mind first, however $y=ax^2+bx+c+\epsilon$ is also linear regression, i.e. $x_2=x, x_1=x^2$ and $y=ax_1+bx_2+c+\epsilon$. It's just we use polynomial features. The data (target) can be of parabolic nature but it can still be estimated via linear regression if you use polynomial features. High bias occurs when you use overly simplistic model compared to the data; not specifically when you use the model $y=ax+b+\epsilon$.

Unbiased estimator is a slightly different concept. If an estimator, say $\hat{\theta}$ for a variable $\theta$ is unbiased, then we have $E[\hat{\theta}]=\theta$. A very simple unbiased estimator is the mean; it is also unbiased since if $\hat{\theta}=\mu$ then the expected value of it will equal to the mean: $E[\hat{\theta}]=E[\mu]=\mu=E[\theta]$. Therefore, let alone OLS, using just the mean is an unbiased estimation technique. So, having an unbiased estimator doesn't mean that your estimator fits well to your data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.