I'm struggling to grasp the concept of bias in the context of linear regression analysis.

  • What is the mathematical definition of bias?

  • What exactly is biased and why/how?

  • Illustrative example?


Bias is the difference between the expected value of an estimator and the true value being estimated. For example the sample mean for a simple random sample (SRS) is an unbiased estimator of the population mean because if you take all the possible SRS's find their means, and take the mean of those means then you will get the population mean (for finite populations this is just algebra to show this). But if we use a sampling mechanism that is somehow related to the value then the mean can become biased, think of a random digit dialing sample asking a question about income. If there is positive correlation between number of phone numbers someone has and their income (poor people only have a few phone numbers that they can be reached at while richer people have more) then the sample will be more likely to include more people with higher incomes and therefore the mean income in the sample will tend to be higher than the population income.

The are also some estimators that are naturally biased. The trimmed mean will be biased for a skewed population/distribution. The standard variance is unbiased for SRS's if either the population mean is used with denominator $n$ or the sample mean is used with denominator $n-1$.

Here is a simple example using R, we generate a bunch of samples from a normal with mean 0 and standard deviation 1, then compute the average mean, variance, and standard deviation from the samples. Notice how close the mean and variance averages are to the true values (sampling error means they won't be exact), now compare the mean sd, it is a biased estimator (though not hugely biased).

> tmp.data <- matrix( rnorm(10*1000000), ncol=10 )
> mean( apply(tmp.data, 1, mean) )
[1] 0.0001561002
> mean( apply(tmp.data, 1, var) )
[1] 1.000109
> mean( apply(tmp.data, 1, sd) )
[1] 0.9727121

In regression we can get biased estimators of slopes by doing stepwise regression. A variable is more likely to be kept in a stepwise regression if the estimated slope is further from 0 and more likely to be dropped if it is closer to 0, so this is biased sampling and the slopes in the final model will tend to be further from 0 than the true slope. Techniques like the lasso and ridge regression bias slopes towards 0 to counter the selection bias away from 0.

  • $\begingroup$ SRS? $\text{ }$ $\endgroup$ – cardinal Jul 30 '11 at 14:03
  • $\begingroup$ @cardinal Simple Random Sample. $\endgroup$ – whuber Jul 30 '11 at 15:34
  • 1
    $\begingroup$ @whuber: Wow. While the abbreviation makes sense, I don't recall having come across it in any more formal settings. Are there particular subfields or applied areas where that is a "standard" initialism? $\endgroup$ – cardinal Jul 30 '11 at 15:56
  • 1
    $\begingroup$ @cardinal See en.wikipedia.org/wiki/Simple_random_sample $\endgroup$ – whuber Jul 30 '11 at 16:07
  • 1
    $\begingroup$ (+1) @whuber's edit was helpful in clarifying this answer. $\endgroup$ – cardinal Jul 30 '11 at 16:08

Bias means that the expected value of the estimator is not equal to the population parameter.

Intuitively in a regression analysis, this would mean that the estimate of one of the parameters is too high or too low. However, ordinary least squares regression estimates are BLUE, which stands for best linear unbiased estimators. In other forms of regression, the parameter estimates may be biased. This can be a good idea, because there is often a tradeoff between bias and variance. For example, ridge regression is sometimes used to reduce the variance of estimates when there is collinearity.

A simple example may illustrate this better, although not in the regression context. Suppose you weigh 150 pounds (verified on a balance scale that has you in one basket and a pile of weights in the other basket). Now, you have two bathroom scales. You weigh yourself 5 times on each.

Scale 1 gives weights of 152, 151, 151.5, 150.5 and 152.

Scale 2 gives weights of 145, 155, 154, 146 and 150.

Scale 1 is biased, but has lower variance; the average of the weights is not your true weight. Scale 2 is unbiased (the average is 150), but has much higher variance.

Which scale is "better"? It depends on what you want the scale to do.

  • 1
    $\begingroup$ Although the definition of bias is correct, I fear that the examples confuse it with inaccuracy, which is something altogether different! Bias is a property of a statistical procedure (an estimator) whereas accuracy is a property of a measurement process. (-1). $\endgroup$ – whuber Jul 30 '11 at 15:33
  • 1
    $\begingroup$ @whuber: Yes, I agree with that. And, I still think that, even so, it is necessary to make clear the difference between mathematical expectation and a sample average, as they relate to bias. $\endgroup$ – cardinal Jul 30 '11 at 16:12
  • 1
    $\begingroup$ No, I was not trying to say anything about "inaccuracy" (which is awfully hard to define) but about "variance". One scale is unbiased, the other scale has low variance. I did not use the word "accurate" or "accuracy". A scale which tends to estimate your weight too high (or too low) is biased. $\endgroup$ – Peter Flom Jul 30 '11 at 16:31
  • 1
    $\begingroup$ But this sense of "bias" is just a synonym for inaccurate; it is not the same as the definition you gave in the first line. Moreover, as @cardinal points out, the example also confounds an expectation with the mean of a particular sample. $\endgroup$ – whuber Jul 30 '11 at 16:38
  • 3
    $\begingroup$ I agree with @whuber here. In the (proper) sense of bias that the OP is asking about, it is not the scale that is biased or unbiased, but rather whatever estimate of your weight that you derive from its measurements! $\endgroup$ – cardinal Jul 30 '11 at 16:50

In Linear regression analysis, bias refer to the error that is introduced by approximating a real-life problem, which may be complicated, by a much simpler model. In simple terms, you assume a simple linear model such as y*=(a*)x+b* where as in real life the business problem could be y = ax^3 + bx^2+c.

It can be said that the expected test MSE(Mean squared error) from a regression problem can be decomposed as below. E(y0 - f*(x0))^2 = Var(f*(x0)) + [Bias(f*(x0))]^2 + Var(e)

f* -> functional form assumed for linear regression model y0 -> original response value recorded in test data x0 -> orginal predictor value recorded in test data e -> irreducible error So, the goal is selecting a best method in arriving a model that achieves low variance and low bias.

Note: An Introduction to Statistical Learning by Trevor Hastie & Robert Tibshirani has a good insights on this topic

  • 4
    $\begingroup$ This is often referred to by something like "model mis-specification error" in order not to confuse it with the standard definition of bias given in the accepted answer. Otherwise it would be impossible to make sense of the (correct) assertion that OLS is an unbiased estimator of the coefficients of the regressors. $\endgroup$ – whuber Feb 1 '19 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.