What intuitively is "bias"?

Question

I'm struggling to grasp the concept of bias in the context of linear regression analysis.

• What is the mathematical definition of bias?

• What exactly is biased and why/how?

• Illustrative example?

Answer 1

Bias is the difference between the expected value of an estimator and the true value being estimated. For example the sample mean for a simple random sample (SRS) is an unbiased estimator of the population mean because if you take all the possible SRS's find their means, and take the mean of those means then you will get the population mean (for finite populations this is just algebra to show this). But if we use a sampling mechanism that is somehow related to the value then the mean can become biased, think of a random digit dialing sample asking a question about income. If there is positive correlation between number of phone numbers someone has and their income (poor people only have a few phone numbers that they can be reached at while richer people have more) then the sample will be more likely to include more people with higher incomes and therefore the mean income in the sample will tend to be higher than the population income.

The are also some estimators that are naturally biased. The trimmed mean will be biased for a skewed population/distribution. The standard variance is unbiased for SRS's if either the population mean is used with denominator $n$ or the sample mean is used with denominator $n-1$.

Here is a simple example using R, we generate a bunch of samples from a normal with mean 0 and standard deviation 1, then compute the average mean, variance, and standard deviation from the samples. Notice how close the mean and variance averages are to the true values (sampling error means they won't be exact), now compare the mean sd, it is a biased estimator (though not hugely biased).

> tmp.data <- matrix( rnorm(10*1000000), ncol=10 )
> mean( apply(tmp.data, 1, mean) )
[1] 0.0001561002
> mean( apply(tmp.data, 1, var) )
[1] 1.000109
> mean( apply(tmp.data, 1, sd) )
[1] 0.9727121

In regression we can get biased estimators of slopes by doing stepwise regression. A variable is more likely to be kept in a stepwise regression if the estimated slope is further from 0 and more likely to be dropped if it is closer to 0, so this is biased sampling and the slopes in the final model will tend to be further from 0 than the true slope. Techniques like the lasso and ridge regression bias slopes towards 0 to counter the selection bias away from 0.

Answer 2

Bias means that the expected value of the estimator is not equal to the population parameter.

Intuitively in a regression analysis, this would mean that the estimate of one of the parameters is too high or too low. However, ordinary least squares regression estimates are BLUE, which stands for best linear unbiased estimators. In other forms of regression, the parameter estimates may be biased. This can be a good idea, because there is often a tradeoff between bias and variance. For example, ridge regression is sometimes used to reduce the variance of estimates when there is collinearity.

A simple example may illustrate this better, although not in the regression context. Suppose you weigh 150 pounds (verified on a balance scale that has you in one basket and a pile of weights in the other basket). Now, you have two bathroom scales. You weigh yourself 5 times on each.

Scale 1 gives weights of 152, 151, 151.5, 150.5 and 152.

Scale 2 gives weights of 145, 155, 154, 146 and 150.

Scale 1 is biased, but has lower variance; the average of the weights is not your true weight. Scale 2 is unbiased (the average is 150), but has much higher variance.

Which scale is "better"? It depends on what you want the scale to do.

Answer 3

In Linear regression analysis, bias refer to the error that is introduced by approximating a real-life problem, which may be complicated, by a much simpler model. In simple terms, you assume a simple linear model such as y*=(a*)x+b* where as in real life the business problem could be y = ax^3 + bx^2+c.

It can be said that the expected test MSE(Mean squared error) from a regression problem can be decomposed as below. E(y0 - f*(x0))^2 = Var(f*(x0)) + [Bias(f*(x0))]^2 + Var(e)

f* -> functional form assumed for linear regression model y0 -> original response value recorded in test data x0 -> orginal predictor value recorded in test data e -> irreducible error So, the goal is selecting a best method in arriving a model that achieves low variance and low bias.

Note: An Introduction to Statistical Learning by Trevor Hastie & Robert Tibshirani has a good insights on this topic