# Different usage of the term "Bias" in stats/machine learning

I think I've seen about 4 different usages of the word "bias" in stats/ML, and all these usages seem to be non-related. I just wanted to get clarification that the usages are indeed non-related.

Here are the 4 I've seen:

(1) "Bias"-variance tradeoff: Here, bias is used to characterize the model error due to simplified assumptions of the model, e.g., using linear regression when linear regression is not complex enough to capture the trends of the data.

(2) "Bias" of an estimator: the difference between the expectation of an estimator and its true value.

(3) "Bias" as in the offset term: e.g., in simple linear regression with $$y = \hat{\beta}_1x + \hat{\beta}_0$$, I sometimes see the offset term, $$\hat{\beta}_0$$, referred to as the "bias" term. e.g., in Role of the bias term in regression

(4) "Biased" data: I think this might be similar to (1), but here it's referring to the data and not the model. I typically see this in experimental studies where the experiment was conducted in a setting that captures a limited portion of the trend. e.g., you could have some response that's sinusoidal, $$e.g., y = \sin(x)$$, and the experiment was only performed for $$x \in [0, \pi/2]$$, which is a monotonically increasing period.

I think these are the 4 usages that I've seen, but if there are others, please do share. The usage of "bias" in these 4 settings are un-related, correct? It's a little confusing to see the same term show up many times in different settings in the same field.

• 1 and 2 seem related, no? And related to 4 there's learning of social bias implicit in the data. Mar 1, 2021 at 14:48
• Bravo for the question! Mar 4, 2021 at 9:31

I will give you a run-down of the terminology used in statistics, which I think is sensible terminology. Cases (1) and (2) do refer to bias in the usual statistical sense, (4) refers to something closely related, and (3) is just misleading renaming of an object that already has a perfectly sensible name. Just to quickly dispose of (3), I note that the term $$\beta_0$$ in regression is called the "intercept" term, not the "bias". Unless there are good contextual reasons to refer to it as the "bias term", that language is highly misleading. (The link you give for this usage is just a CV.SE question that was closed for lack of clarity, so not really evidence of widespread usage. I have never seen this term in the regression referred to as the bias term.)

Estimator bias and the "bias-variance trade-off": When we "fit" a statistical model we are essentially just estimating the unknown parameters in that model. As you note in case (2), the bias of an estimator is defined as the difference between the expected value of the estimator and the value of the parameter it is estimating:

$$\text{Bias}(\hat{\theta}, \theta) = \mathbb{E}(\hat{\theta}) - \theta. \quad \$$

When looking at the performance of estimators we often measure this by the mean squared error, which is the expected value of the squared deviation of the estimator from the value it is estimating:

$$\text{MSE}(\hat{\theta}, \theta) = \mathbb{E} \Big( (\hat{\theta}-\theta)^2 \Big).$$

One of the properties of the mean squared error is that it can be decomposed as:

$$\quad \quad \ \ \text{MSE}(\hat{\theta}, \theta) = \mathbb{V}(\hat{\theta}) + \text{Bias}(\hat{\theta}, \theta) ^2.$$

If we examine a model for an observable value $$y = f(x, \theta) + \varepsilon$$ composed of a regression term and an error term, we likewise have:

$$\mathbb{E} \Big( (y-f(x, \hat{\theta}))^2 \Big) = \mathbb{V}(f(x,\hat{\theta})) + \text{Bias}(f(x,\hat{\theta}), f(x,\theta)) ^2 + \sigma_\varepsilon^2.$$

Now, if we examine the class of estimators with some fixed mean squared error, we can see that there must be a trade-off between the bias and variance in these estimators --- a lower bias corresponds to a higher variance and vice versa. Often when we have competing methods of estimating parameters (i.e., alternative ways to "fit" the model) we focus on those that have the optimal mean squared error, and within this class we see that there is a choice between methods that have higher bias but lower variance, and methods that have low (or no) bias but have higher variance.

As you point out in your case (1), it is common in discussions of machine-learning to refer to a general "bias-variance trade-off" in the choice of model fitting methods and the use of training data. While this discussion is often quite broad and esoteric, ultimately it derives from the statistical decomposition shown here. It is therefore referring to "bias" in the standard statistical sense. Consequently, cases (1) and (2) both refer to bias in its usual statistical definition.

Informative sampling (so-called "biased" data): In discussions of sampling you might sometimes run across references to "bias" in the data or sampling method. Statisticians generally do not use this language (except sometimes as shorthand) because they recognise that bias is a property of an estimator, and so it only occurs from the combination of a sampling method and an inference method. When data gives information in a non-standard way we say that it is an "informative" sampling method and we try to take this into account in our inferences. If the inference method does not properly take account of this information then this gives us a biased estimator, but the bias comes from the combination of the sampling method and the inference method --- it is not the data per se that is biased.

This area of statistics is quite subtle, so I will describe it through an example. Suppose you have a small community with ten children, and you want to know the average number of children per family (only for the families with at least one child). Suppose that of these ten chidren, nine are from the same family (all siblings) and the one is an only child (no siblings), which means that the true average is five. Suppose you sample all the children and ask each of them how many siblings they have, then use this data to estimate the average number of children per family. A naive estimator would be to take the average number of siblings per child and then add one to get an estimate of the average number of children per family. If you use that inference method, you get a substantial overestimate:

$$\hat{\theta} = \tfrac{1}{10} (9 \cdot 9 + 1 \cdot 0) + 1 = 8.1+1 = 9.1.$$

The problem here is that we are sampling over children rather than over families, and so we are more likely to select a child from a larger family; the average over all children gives proportionately more weight to families of larger size. This is an example of "informative" sampling method where the naive estimator leads to large bias in estimating the true quantity of interest. (The technical name for this type of sampling is probability proportional to size (PPS) sampling.) Note here that it is the use of a particular type of (erroneous) estimator that leads to the bias. If we take account of the fact that we are using PPS sampling, and account for this in our estimator, we can get rid of this bias.

As you can see from this example, it is not quite right to say that the data themselves are biased --- the sampling method is unusual, and it leads us to substantial bias if we treat it as if it were giving us direct information on the quantity of interest, but the bias occurs from our failure to account for the nature of the sampling mechanism in our estimator. In statistical discussion we use the term "informative sampling" to describe this kind of sampling mechanism, but "bias" remains a property of the estimator. Consequently, whether or not we have bias is determined by the combination of the sampling method and the estimator.

They all refer to something being "non-neutral", but, apart from that, I wouldn't say they are related. To my understanding, (2) refers to the computational method, irrespective of the data. For example MLE vs. unbiased estimator of the variance, whether you use $$N$$ or $$(N-1)$$ as the denominator. (4), on the other hand, is about the sampling process.

Regarding (3), based on my professional experience and acquaintance with machine learning history, I believe the usage of "bias" with the meaning of "intercept" comes from electronics: The early research on what we today call "machine learning", in the 1950's-60's, often involved building specialised hardware or, later, simulating that hardware in computers. "Bias" in electronics means the intentional shift of the operating voltage away from zero, in order to achieve desired response characteristics (typically: linearity) of a component, like transistor or vacuum tube. This is probably also the reason why the intercept in machine learning is often denoted by $$b$$ and the predictor coefficients ("weights" in neural network terminology) by $$\textbf{w}$$.

• Ah ok. That would make sense then. Thanks for sharing!
– matt
Mar 2, 2021 at 13:45

## (2)

What is bias? As you correctly defined in (2), bias is the difference between the estimator and its true value in expectation.

## (4)

This is an application of the definition. Making things a bit explicit here can help. What we want to estimate is:

E[sin(x) | full_period]

E[sin(x) | quarter_period]

Clearly:

E[sin(x) | full_period] != E[sin(x) | quarter_period]

in general.

Thus our estimator is wrong. It's wrong because we incorrectly sampled on a different set of conditions. But it still fits the definition of bias.

## (3) and (1)

(3) is interesting. I am not sure why historically the term of 'bias' originated in linear regression. If I simulated data from a linear regression model with a non-zero intercept and then built a linear regression model from its output data, clearly my non-zero 'bias' term is what we want--so it's not biased according to our definition of bias. This suggests it does indeed reflect a different meaning.

It seems by 'bias' in this context, we are really trying to hint at this idea of how well the model is influenced by data. That is, the larger we make our intercept, the less impact the data has on our predictions. This would also capture this notion .

## Research Paper

I also found this research paper while answering this question (have not read it) that attempts to distinguish the bias definitions used in machine clearing:

https://arxiv.org/pdf/2004.00686.pdf

• Is there significance to the nested expectation in E[E[sin(x) | full_period]] or was that a typo? Is E[sin(x) | full_period] a random variable? Mar 1, 2021 at 16:04
• (1) is not too interesting. If $\hat{\beta}$ is an estimator which may be biased, $\hat{Y}$ is also an estimator - a function of $\hat{\beta}$ - which may also be biased. You would simply measure the bias on a different scale. The parameters may be biased on the parameter scale, but the predictions unbiased on the response scale. Hence "bias-variance" tradeoff in the context of MSE. Mar 1, 2021 at 16:05
• @student010101 My intent was emphasize E[sin(x) | full_period] as a random variable (which changes based on what is being conditioned on). But I think that made it more confusing than necessary. I have removed it.
– matt
Mar 1, 2021 at 17:10

They all mean a systematic (as opposed to a random) deviation from a target, when repeatedly applying a (sampling and/or estimation) procedure. The target differs in the mentioned examples:

1. Target is value of the response in a sample from the target population, that was not part of the training sample.

2. Target is true value of the estimated quantity. (Also goes for point 1. actually)

3. Target is zero.

4. Biased data should read biased sample. Target is the (multivariate) distribution of values in the target population.

To me this is an excellent question, I would be very proud if I asked it. I will not analyze the bias in bias-variance-trade-off in here just, just the bias function instead.

Bias is often related to any of these:

• sampling method
• estimator
• model
• inference method

## Sampling method

Both quality and quantity sampling may influence the later bias, but sampling method itself will not produce the bias.

In short the $$N$$ -- number of elements in your sample or the $$n$$ -- number of samples represent the quantity part and quality would be either of the techniques:

• random sampling
• stratified sampling
• systematic sampling

It depends on a problem what kind of a sampling we choose, but again sampling method may be a cause for a bias, not the instant creator of the bias, because data is innocent.

## Estimator

Estimator and how it relates to a model I don't understand quite, but I will shoot proudly and congruently and check yor comments later:

• Estimator is parameter estimator and can show the bias in parameter estimation
• Model is what estimates the prediction and can show the bias of a prediction

We can call estimator -- the decision rule for selecting parameters.

I learned that there are biased and unbiased estimators. The bias (or bias function) of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated. We usually use thetas for general parameters so:

$$\text{Bias}(\hat{\theta}, \theta) = \mathbb{E}(\hat{\theta}) - \theta. \quad \$$

Let's say bias can also be measured with respect to the median, rather than the mean (expected value), in which case one distinguishes median-unbiased from the usual mean-unbiasedness property.

An estimator with zero bias is called unbiased. In statistics, "bias" is an objective property of an estimator super related to the term called degrees of freedom. At this moment my understanding is: The more degrees of freedom, the more bias, the more estimation errors we have.

## Model

There are many ways you can define what is a model, but in here I will just analyze the models as being able to predict the feature of interest. In that respect the bias will tell us how off from the true value of the prediction we are. I will use this image to save some words.

There is another bias in the model, where we say bias exists because hypothesis functions are biased to a particular kind of solution. In other words, bias is inherent to a model (but I would call this simple as the bias of the estimator).

## Inference method

Classical inference method (I will just stick with that) computes probabilities from multiple hypotheses in order to determine their acceptability. This method usually analyses two hypotheses at a time.

This inference provides quantitative information about a sensor observation in the form of a probability. As I see it -- we cannot create bias in here. We may change and adopt to differnt confidence intervals but this will not provide bias or errors.