Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I hear the term bias being thrown around a lot in statistical literature.
For example,
By using mean-wise imputation, we are adding bias to our estimate.
Another example,
The bias-variance trade-off is an important subject when picking models.
Are these the same "bias"?
The term "bias" has a specific definition in the statistical literature (the difference between the expected value of an estimator and the thing being estimated), but that isn't to say it loses its original, more general meaning. Which one is intended will depend on context, and oftentimes you will have a mixture of the two.
I would say the first usage is in general the less precise kind, as data imputation is a method that's used in applied problems where one need not assume that any true value of the parameter even exists. Here it's basically synonymous with "shrunk towards zero."
As far as the second usage is concerned the term bias-variance trade-off does originally derive from the more formal definition of bias, but nonetheless I would still say this refers more to the general "inflexibility" of a model fitting procedure, and not necessarily the question of whether or not an estimated regression function is correct on average.
I agree that this terminology is confusing. Bias has one meaning in both of these contexts: distance from ideal or target values, but the interpretation depends on which space we are talking about. I'll explain what I mean with regard to the two quotes in your question.
By using mean-wise imputation, we are adding bias to our estimate.
This refers to bias in the data space. Mean-wise imputation influences the position of your estimates relative to the target values.
The bias-variance trade-off is an important subject when picking models.
This refers to bias and variance in the parameter space of models. That is, if you trained a stochastic model 1000 times, you could observe bias or variance of the parameter values. A high bias model has consistent parameters, but they differ from an 'optimal' solution. A high variance model will get different values for the parameters each time it's trained.
