7
$\begingroup$

In the Wikipedia article for Minimum Mean Square Error, the Bayesian estimator is referred as "asymptotically efficient" using the similar arguments of Fisher information from frequentist statistics.

Does it even make sense to talk about the "efficiency" (or "asymptotic unbiasedness") of a Bayesian estimator given a credible prior at hand?

$\endgroup$
9
  • 1
    $\begingroup$ Why would it not make sense to talk about the frequentist properties of a Bayesian estimator? Even if someone were to use his Bayesian estimate for purely private and subjective reasons, that need to prevent others to think about if the employed estimation strategy has good properties in repeated samples. Also, can you clarify what you mean by a "credible" prior? $\endgroup$ Commented Dec 4, 2019 at 14:11
  • $\begingroup$ @ChristophHanck Because Bayesian doesn't assume that there is a true value, with respect to what are we going to measure the efficiency? With credible prior I mean a well thought prior by someone with domain experience, not just any prior for the sake of doing Bayes update. $\endgroup$ Commented Dec 4, 2019 at 14:18
  • 1
    $\begingroup$ I do not think that all Bayesians dismiss the notion of a true value, and even if they did, no Bayesian could prevent a frequentist (or anyone who does work with such a notion) from talking about the efficiency or any other frequentist property of such a Bayesian estimator. $\endgroup$ Commented Dec 4, 2019 at 14:43
  • $\begingroup$ I could say that I am a Hitchhikian - let us say, a school that dismisses the notion of a true value - who believes that 42 is the answer to everything (en.wikipedia.org/wiki/42_(number)). You could talk about the bias properties of that estimator, couldn't you :-)? $\endgroup$ Commented Dec 4, 2019 at 15:07
  • 1
    $\begingroup$ Could you give a reference to some Bayesians not assuming existence of a true parameter value? In absence of a true value, I wonder what their estimators are targeting and how they assess the goodness of their estimators. Or perhaps I should post this as a separate question? $\endgroup$ Commented Dec 4, 2019 at 21:22

1 Answer 1

3
+50
$\begingroup$

This is an example of where a Bayesian estimator is examined with respect to its frequentist properties ---i.e., how it behaves over repeated trials conditional on the parameter value. Although Bayesian estimators are derived using a prior distribution over the possible parameter values, ultimately, the estimators are functions of the data (and possibly some hyperparameters) and so it is possible to examine their behaviour conditional on a parameter value. Bayesians also believe that there is a true parameter value, and they have a prior belief about what it is. Nevertheless, in the history of Bayesian statistics, a natural question was whether Bayesian estimators have good freqentist properties --- i.e., how do they perform over repeated trials if we condition on a stipulated parameter value.

Your confusion on this issue appears to be coming from equivocation of two separate issues. You are correct that the Bayesian estimator is formed using a prior distribution over the parameter. This is a reflection of the fact that the true value is unknown, and the Bayesian paradigm applies a prior distribution to describe one's prior belief about the true value. However, the assessment of the frequentist properties of a Bayesian estimator is a separate issue from the formation of the estimator; it is done by conditioning on a stipulated value of the parameter, and then asking how the estimator performed over repeated trials relative to this true value. In particular, when examining consistency, efficiency and unbiasedness, we condition on the value of the parameter --- this is also the case even if we are assessing the properties of a Bayesian estimator.

The good news here is that Bayesian estimators have good frequentist properties, so long as the model (i.e., likelihood function) is specified correctly. In particular, Bayesian estimators can be shown to be "admissible", which means that they are not "dominated" by any other estimator in terms of their efficiency. Since Bayesian estimators have been proven to have good frequentist properties, it is common for Bayesians to feel that they can safely forget about this issue, and work entirely within the Bayesian paradigm. That is quite reasonable, but it is also handy to be able to pop back into the classical paradigm and understand the frequentist assessment of these estimators.


Asymptotic efficiency of a Bayesian estimator: Under broad regularity conditions, if we have a Bayesian estimator $\hat{\theta}_n$ of an unknown parameter $\theta \in \Theta$ that is formed by minimising the posterior MSE, it is possible to show that there is asymptotic convergence:

$$\sqrt{n} (\hat{\theta}_n-\theta) \overset{\rightarrow}{\sim} \text{N} \Big( 0, \frac{1}{I(\theta)} \Big) \quad \quad \quad \text{for all } \theta \in \Theta,$$

where $I$ is the Fisher information. This convergence result holds conditional on each $\theta \in \Theta$, so it is a frequentist result, rather than a marginal result aggregated over a prior. Since this variance in the asymptotic distribution is the Cramér-Rao lower bound, that is sufficient to establish asymptotic efficiency of the estimator. There are of course other estimators that are also asymptotically efficient, but it is nice to know that this Bayesian estimator has that property.

$\endgroup$
7
  • $\begingroup$ So is a Bayesian actually a frequentist with a constrained search space for the parameter? Is it possible to apply CRLB with a prior? $\endgroup$ Commented Dec 5, 2019 at 5:56
  • $\begingroup$ It is possible that one could be motivated to apply Bayesian methods solely by their frequentist properties, but most Bayesians regard the paradigm as much more than this ---e.g., philosophical coherence, etc. What is CRLB? $\endgroup$
    – Ben
    Commented Dec 5, 2019 at 6:47
  • 1
    $\begingroup$ I am not sure that Bayesians can be argued to have a "constrained search space for the parameter" - that would seem to me imply that, for example, it could happen that frequentists allow a mean $\mu$ to be in all of $\mathbb{R}$ while a Bayesian does not. Of course, if a prior assigns zero probability to some subset of $\mathbb{R}$, that would happen, but in general, that would not be recommended, and indeed, typically priors just assign higher a priori probability to some parameters than to others rather than ruling some out completely. $\endgroup$ Commented Dec 5, 2019 at 7:39
  • $\begingroup$ @ChristophHanck It is a soft constraint if there is positive probability and a hard constraint if there is zero probability. You are for some reason dragging the question to irrelevant directions. $\endgroup$ Commented Dec 5, 2019 at 7:55
  • 1
    $\begingroup$ I do not think so. If your question about constraints seeks to distinguish between soft and hard constraints it would be helpful to spell this out. Given the current formulation all my comment does is to try and address your question. $\endgroup$ Commented Dec 5, 2019 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.