7
$\begingroup$

I hear the term bias being thrown around a lot in statistical literature.

For example,

By using mean-wise imputation, we are adding bias to our estimate.

Another example,

The bias-variance trade-off is an important subject when picking models.

Are these the same "bias"?

$\endgroup$
  • 1
    $\begingroup$ There are actually 12 definitions: dictionary.com/browse/bias?s=t $\endgroup$ – StatsStudent Mar 18 '16 at 2:41
  • 1
    $\begingroup$ @StatsStudent those are not definitions of it's statistical meaning $\endgroup$ – Tim Aug 24 '16 at 14:34
  • $\begingroup$ But when you hear "bias," even in the statistical literature, it does not necessarily have a statistical meaning... $\endgroup$ – StatsStudent Aug 27 '16 at 1:01
3
$\begingroup$

The term "bias" has a specific definition in the statistical literature (the difference between the expected value of an estimator and the thing being estimated), but that isn't to say it loses its original, more general meaning. Which one is intended will depend on context, and oftentimes you will have a mixture of the two.

I would say the first usage is in general the less precise kind, as data imputation is a method that's used in applied problems where one need not assume that any true value of the parameter even exists. Here it's basically synonymous with "shrunk towards zero."

As far as the second usage is concerned the term bias-variance trade-off does originally derive from the more formal definition of bias, but nonetheless I would still say this refers more to the general "inflexibility" of a model fitting procedure, and not necessarily the question of whether or not an estimated regression function is correct on average.

$\endgroup$
  • $\begingroup$ Hm - for the first one: the whole point of mean-wise imputation is to maintain the mean of the original sample. Hence both parameters should be the same - so it has no bias.\ $\endgroup$ – user46925 Mar 18 '16 at 11:10
  • $\begingroup$ I'm fairly sure they're talking about the bias of the coefficient estimate and not the bias of the sample mean. Also, the sample mean need not be unbiased when you have missing data. $\endgroup$ – dsaxton Mar 18 '16 at 13:10
2
$\begingroup$

I agree that this terminology is confusing. Bias has one meaning in both of these contexts: distance from ideal or target values, but the interpretation depends on which space we are talking about. I'll explain what I mean with regard to the two quotes in your question.

By using mean-wise imputation, we are adding bias to our estimate.

This refers to bias in the data space. Mean-wise imputation influences the position of your estimates relative to the target values.

The bias-variance trade-off is an important subject when picking models.

This refers to bias and variance in the parameter space of models. That is, if you trained a stochastic model 1000 times, you could observe bias or variance of the parameter values. A high bias model has consistent parameters, but they differ from an 'optimal' solution. A high variance model will get different values for the parameters each time it's trained.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy