I know that bias is the difference between this estimator's expected value and the true value of the parameter being estimated. In classical approach the parameter has one particular true value, meanwhile in Bayesian approach the parameter value (is fixed but) is chosen from a probability distribution.

But is it Bayesian or frequentist concept?

  • $\begingroup$ Maybe it is a statistical concept? (But it seems to be difficult to formalize in a bayesian context). $\endgroup$ – kjetil b halvorsen May 24 '17 at 14:45
  • $\begingroup$ I do not think Bayesian approach has the definition of bias. Next question is: how does the Bayesian evaluate the estimators given no definition of bias? because bias and MSE cannot be used. $\endgroup$ – user158565 May 24 '17 at 16:43
  • $\begingroup$ @a_statistician E.T. Jaynes, who was quite the militant Bayesian, seems to use the standard frequentist definition of bias in his writings. He also uses MSE as the default way to evaluate estimators, unless some problem-specific criterion is known. Jaynes's main quarrel is that reducing bias does not necessary lead to reduced MSE, so he calls it an 'emotionally loaded term', and that 'orthodoxians... are caught in psychosemantic trap of their own making'... $\endgroup$ – juod May 24 '17 at 17:50
  • $\begingroup$ @juod The problem is the definition of bias in Bayesian. I said NO, and you said Jaynes had one. Could you introduce Jaynes's bias? Then we can discuss Jaynes's bias. $\endgroup$ – user158565 May 24 '17 at 18:54
  • $\begingroup$ @a_statistician it seems to me that Jaynes is fine with the concept of one true parameter value, because he introduces bias as $E(\hat\theta) = \theta$, and proceeds to argue only about whether it should be the criterion for estimator choice. But I could certainly be wrong. $\endgroup$ – juod May 24 '17 at 19:54

Suppose there is a model for the data $Y$ that depends on a parameter $\theta$ and, for a particular experiment, there is a true value of the parameter, $\theta_0$. You develop an estimator $\hat\theta = \hat\theta(Y)$, i.e. the estimator is a function of the data $Y$. Then the bias is $$ bias(\hat\theta) = E_{Y|\theta_0}[\hat\theta(Y) - \theta_0] $$ where the expectation is taken with respect to the randomness of the data $Y$ for the given true value of the parameter $\theta_0$ (and the subscript on the expectation attempts to make this explicit). As we are talking about an expectation over possible realizations of data, this is a frequentist concept.

In the description above, I have not mentioned how the estimator arises. This estimator could be a method of moments, maximum likelihood, Bayes, or something else estimator. Thus, the concept of bias of an estimator is frequentist, but the estimator itself could arise from a Bayesian analysis.

  • 1
    $\begingroup$ This is still ambiguous. The whole idea of there being a true, fixed theta is itself very non-Bayesian. For strict Bayesian, the parameter itself is random. All we have is prior uncertainty about theta, and posterior uncertainty about theta. If you don't have a true, fixed theta, you can't compute the bias as given above. You need to average that bias over the distribution of theta itself. $\endgroup$ – AOGSTA Mar 10 '19 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.