2
$\begingroup$

I'm confused about the usage of the term "prior" vs "bias". They often seem to be synonymous?

Example 1: I'm estimating the probability for heads of a coin, and use a Beta(20, 20) prior. Then you could say that my estimate will be biased towards a fair coin, yes? So I'm making a biased estimate. Is this an appropriate use of the two terms?

Example 2: I'm developing a meta-learning system to detect faces of mammals. Once the meta-learning is finished, the system can be fine-tuned on 5 gorilla faces to detect gorillas with 95% accuracy, or on 50 rat faces to detect rats with 95% accuracy. So my system has a built-in prior towards gorillas. Or is it a bias towards gorillas? Are both terms equally appropriate?

$\endgroup$
1
$\begingroup$

Bias is a measure of the model's in-sample fitting ability. Generally, a more flexible model will have a lower bias (ie it fits the data well). The problem with low-bias models is that they can fit the data too well (ie. they start fitting the noise in the data too).

This where the need of adding some discipline to the model arises. So, in some cases like lasso or ridge regression, people regularize the model by adding a penalty to a non-zero parameter estimate. This is the same as adding a prior for the model parameter (for example, ridge regression is equivalent to linear regression with a normal prior, of mean zero and certain variance, for parameters).

A prior is used to incorporate null hypothesis/expert opinion/theoretical insights on the distribution of model parameters. This is often used to ensure that a very flexible model (which has an inclination to fit the data well) doesn't come up with estimates which "don't make sense" ie. are driven by noise in the sample (for example, if my model estimates the probability of "getting a head" to be 1, it's possibly because I have a sample which doesn't represent the population well. So, I make it hard for the model to come up with such an estimate).

In fact, it can be shown, at least in the simple case of the linear regression, that adding a prior is equivalent to adding additional data generated under our null hypothesis (so in your example we are adding a few tosses of a fair coin) to the sample collected from the external world. So, you are right, a prior does bias the model towards a certain outcome. But this is often seen as a disciplining bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.