Difference between Bias and Error?

In statistics, what is the difference between Bias and Error?

You can say, Bias is a type of error? or Bias is an error with some tendency?

• I think they are different things. – SmallChess Feb 2 '15 at 10:24
• Bias relates to expectation while error measures deviation in the sample to population. – SmallChess Feb 2 '15 at 10:25
• Bias doesn't go to 0 asymptotically, – user603 Feb 2 '15 at 11:29
• I have often seen "bias" used to describe differences from expectation that are systematic, and "error" to describe differences that are (or appear) random. – D L Dahly Feb 2 '15 at 14:25
• I learned that BIAS is how far off on the average the model is from the true. – Darwin PC Feb 12 '15 at 3:02

We can talk about the error of a single measurement, but bias is the average of errors of many repeated measurements. Bias is a statistical property of the error of a measuring technique. Sometimes the term "bias error" is used as opposed to "root-mean-square error".

• RMS error inevitably includes any bias. – Nick Cox Feb 2 '15 at 11:39

The term error appears in several related (but not identical) contexts throughout science in general and statistical science in particular.

Error still carries the flavour of mistake (something erroneous), at least in the context of measurement error and particularly when scientists are thinking about their data. But its primary meaning in statistical science has long since been simply that of more or less uncontrolled variation (something erratic or errant). Sampling error, for example, refers to sampling variation, the uncontrolled and uncontrollable fact that different samples, responsibly taken, will include different data; hence in general any statistics (such as means, correlations, fraction blue) based on those samples will differ from sample to sample.

In simple regression-type models, error refers to individual disturbances in specifications such as

response variable $=$ function of predictors $+$ stochastic error

and error can refer more generally to the conditional distribution of the response variables given the predictors.

Bias refers to the difference between the true or correct value of some quantity and a measurement or estimate of that quantity. In principle it cannot be calculated therefore unless that true or correct value is known, although this problem bites to varying degrees.

• In the simplest kind of problem, the true value is known (as when the centre of a target is visible and the distance of a shot from the centre can be measured; this is a common analogy) and bias is then usually calculated as the difference between the true value and the mean (or occasionally some other summary) of measurements or estimates.

• In other problems, some careful method is regarded as the state of the art and so yielding the best possible measurements, and so other methods are regarded as more or less biased according to their degree of systematic departure from the best method (in some fields termed a gold standard).

• In yet other problems, we have one or more methods all deficient to some degree and assessment of bias is then difficult or impossible. It is then tempting, or possibly even natural, to change the question and judge truth according to consistency between methods.

The two terminologies can be made consistent with the idea that systematic measurement errors have non-zero means (hence their summary quantifies bias) and random errors have zero mean. (Equivalently, that is how we label error as systematic or random.)

In mathematical statistics, standard analyses analyse whether particular estimators are biased in small samples, asymptotically, etc,, either in general or under particular circumstances.

This sketch at times implies that error is defined additively, so that

measured value $=$ true value $+$ error

but that is just the simplest situation. Nothing here rules out the idea that error may be multiplicative rather than additive, or defined on more complicated scales (e.g. in measuring proportions or percents, error may be better considered on something like a logit scale).

Comments on erroneous and erratic here were inspired by discussions in Jeffreys, Harold. 1939/1948/1961. Theory of probability. London: Oxford University Press.

The difference between the two is not only semantic, one can also express the difference in a formula: the bias-variance-tradeoff.

The following is the bias-variance decomposition as in Elements of Statistical Learning or the wikipedia page on bias-variance tradeoff:

$$\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + \text{Bias}^2(\hat{\theta},\theta).$$

Where $\hat{\theta}$ is the estimator for $\theta$, $\text{MSE}(\hat{\theta}) = \mathbb{E}(\hat{\theta}-\theta)^2$ is the mean square error, $\text{Var}(\hat{\theta})=\mathbb{E}(\hat{\theta}-\mathbb{E}\hat{\theta})^2$ is the variance of $\hat{\theta}$ and $\text{Bias}^2(\hat{\theta},\theta)= (\mathbb{E}\hat{\theta} - \theta)^2$ is the bias (systematic deviation) of the estimator.

Form this identity we can see that in the context of estimators,

• bias is an error because it is a component of the mean square error.
• not every error is a bias (unfortunately)
• (this is not related to the question) there might be biased estimators that can have a lower MSE than unbiased estimators although it is a nice property for an estimator to be unbiased.

What I present here is about the terms error and bias for estimators but I think the principles hold true for the words as they are used in statistics in general:

One can decompose error into a systematic and an unsystematic component. Bias is a name for the systematic error.

To put it succinctly, bias is the difference of the expected value of your estimate (denote as $\hat{\theta}$) with the true value of what you are estimating (denote as $\theta$).

$$E[\hat{\theta}] - \theta$$

Error is the difference of your estimate with the true value of what you are estimating.

$$\hat{\theta} - \theta$$

You can have a fantastic estimator which is unbiased, but still have error because your observed value of the estimator didn't get it exactly right.