# Example of how Bayesian Statistics can estimate parameters that are very challenging to estimate through frequentist methods

Bayesian statisticians maintain that "Bayesian Statistics can estimate parameters that are very challenging to estimate through frequentist methods". Does the following quote taken from this SAS documentation says that same thing?

It provides inferences that are conditional on the data and are exact, without reliance on asymptotic approximation. Small sample inference proceeds in the same manner as if one had a large sample. Bayesian analysis also can estimate any functions of parameters directly, without using the "plug-in" method (a way to estimate functionals by plugging the estimated parameters in the functionals).

I saw a similar statement in some textbook but do not recall where. Can anyone please explain this to me with an example?

• What is the probability that the sun will rise tomorrow? en.wikipedia.org/wiki/Sunrise_problem This example might be more trivial than you were hoping for – Hugh Oct 30 '16 at 23:40
• Can you put the quote directly into your question? The title seems unrelated to the second bullet point. – Hugh Oct 30 '16 at 23:44
• The statement in that quote is not great in that (a) "exact" does not mean anything there and (b) the plug-in criticism only applies when one considers the full posterior and not another estimate, plus depending on the loss function chosen for the estimation. See this other question for some answers. – Xi'an Oct 31 '16 at 6:06

1. "Frequentism" is an approach to inference that is based on the frequency properties of the chosen estimators. This is a vague notion in that it does not even state that the estimators must converge and if they do under how they must converge. For instance, unbiasedness is a frequentist notion but it cannot hold for any and every function [of the parameter $\theta$] of interest since the collection of transforms of $\theta$ that allow for an unbiased estimator is very restricted. Further, a frequentist estimator is not produced by the paradigm but must first be chosen before being evaluated. In that sense, a Bayesian estimator is a frequentist estimator if it satisfies some frequentist property.
2. The inference produced by a Bayesian approach is based on the posterior distribution, represented by its density $\pi(\theta|\mathfrak{D})$. I do not understand how the term "exact" can be attached to $\pi(\theta|\mathfrak{D})$.It is uniquely associated with a prior distribution $\pi(\theta)$ and it is exactly derived by Bayes theorem. But it does not return exact inference in that the point estimate is not the true value of the parameter $\theta$ and it produces exact probability statements only within the framework provided by the pair prior x likelihood. Changing one term in the pair does modify the posterior and the inference, while there is no generic argument for defending a single prior or likelihood.
4. From a computational perspective, it is true that a Bayesian approach may often return exact or approximate answers in cases when a standard classical approach fails. This is for instance the case for latent [or missing] variable models$$f(x|\theta)=\int g(x,z|\theta)\,\text{d}z$$where $g(x,z|\theta)$ is a joint density for the pair $(X,Z)$ and where $Z$ is not observed, Producing estimates of $\theta$ and of its posterior by simulation of the pair $(\theta,\mathfrak{Z})$ may prove much easier than deriving a maximum likelihood [frequentist?] estimator. A practical example of this setting is Kingman's coalescent model in population genetics, where the evolution of populations from a common ancestor involves latent events on binary trees. This model can be handled by [approximate] Bayesian inference through an algorithm called ABC, even though there also exist non-Bayesian software resolutions.
6. A final point is that, for point estimation, the Bayesian approach may well produce plug-in estimates. For some loss functions that I called intrinsic losses, the Bayes estimator of the transform $\mathfrak{h}(\theta)$ is the transform $\mathfrak{h}(\hat\theta)$ of the Bayes estimator of $\theta$.