Bayesian statisticians maintain that "Bayesian Statistics can estimate parameters that are very challenging to estimate through frequentist methods". Does the following quote taken from this SAS documentation says that same thing?

It provides inferences that are conditional on the data and are exact, without reliance on asymptotic approximation. Small sample inference proceeds in the same manner as if one had a large sample. Bayesian analysis also can estimate any functions of parameters directly, without using the "plug-in" method (a way to estimate functionals by plugging the estimated parameters in the functionals).

I saw a similar statement in some textbook but do not recall where. Can anyone please explain this to me with an example?

  • $\begingroup$ What is the probability that the sun will rise tomorrow? en.wikipedia.org/wiki/Sunrise_problem This example might be more trivial than you were hoping for $\endgroup$ – Hugh Oct 30 '16 at 23:40
  • $\begingroup$ Can you put the quote directly into your question? The title seems unrelated to the second bullet point. $\endgroup$ – Hugh Oct 30 '16 at 23:44
  • 1
    $\begingroup$ The statement in that quote is not great in that (a) "exact" does not mean anything there and (b) the plug-in criticism only applies when one considers the full posterior and not another estimate, plus depending on the loss function chosen for the estimation. See this other question for some answers. $\endgroup$ – Xi'an Oct 31 '16 at 6:06

I have objections with that quote:

  1. "Frequentism" is an approach to inference that is based on the frequency properties of the chosen estimators. This is a vague notion in that it does not even state that the estimators must converge and if they do under how they must converge. For instance, unbiasedness is a frequentist notion but it cannot hold for any and every function [of the parameter $\theta$] of interest since the collection of transforms of $\theta$ that allow for an unbiased estimator is very restricted. Further, a frequentist estimator is not produced by the paradigm but must first be chosen before being evaluated. In that sense, a Bayesian estimator is a frequentist estimator if it satisfies some frequentist property.
  2. The inference produced by a Bayesian approach is based on the posterior distribution, represented by its density $\pi(\theta|\mathfrak{D})$. I do not understand how the term "exact" can be attached to $\pi(\theta|\mathfrak{D})$.It is uniquely associated with a prior distribution $\pi(\theta)$ and it is exactly derived by Bayes theorem. But it does not return exact inference in that the point estimate is not the true value of the parameter $\theta$ and it produces exact probability statements only within the framework provided by the pair prior x likelihood. Changing one term in the pair does modify the posterior and the inference, while there is no generic argument for defending a single prior or likelihood.
  3. Similarly, other probability statements like “the true parameter has a probability of 0.95 of falling in a 95% credible interval” found in the same page of this SAS documentation has a meaning relative to the framework of the posterior distribution but not in absolute value.
  4. From a computational perspective, it is true that a Bayesian approach may often return exact or approximate answers in cases when a standard classical approach fails. This is for instance the case for latent [or missing] variable models$$f(x|\theta)=\int g(x,z|\theta)\,\text{d}z$$where $g(x,z|\theta)$ is a joint density for the pair $(X,Z)$ and where $Z$ is not observed, Producing estimates of $\theta$ and of its posterior by simulation of the pair $(\theta,\mathfrak{Z})$ may prove much easier than deriving a maximum likelihood [frequentist?] estimator. A practical example of this setting is Kingman's coalescent model in population genetics, where the evolution of populations from a common ancestor involves latent events on binary trees. This model can be handled by [approximate] Bayesian inference through an algorithm called ABC, even though there also exist non-Bayesian software resolutions.
  5. However, even in such cases, I do not think that Bayesian inference is the only possible resolution. Machine-learning techniques like neural nets, random forests, deep learning, can be classified as frequentist methods since they train on a sample by cross-validation, minimising an error or distance criterion that can be seen as an expectation [under the true model] approximated by a sample average. For instance, Kingman's coalescent model can also be handled by non-Bayesian software resolutions.
  6. A final point is that, for point estimation, the Bayesian approach may well produce plug-in estimates. For some loss functions that I called intrinsic losses, the Bayes estimator of the transform $\mathfrak{h}(\theta)$ is the transform $\mathfrak{h}(\hat\theta)$ of the Bayes estimator of $\theta$.
  • $\begingroup$ The answer is good as far as it goes. I object to point #5, since it attributes to ML methods a superior performance which has yet to be justified by theory. Also what does "... true model ..." even mean? No doubt, these methods are popular, but that popularity is typically justified by their ability "to scale." Unfortunately, many of the diagnostic insights offered by non-ML Bayesian and frequentist methods are lost when such approaches are used. In particular, cross-validation can yield higher error rates than other techniques See Efron, 1983, 1986, JASA. $\endgroup$ – Jan Galkowski May 13 '17 at 14:40
  • $\begingroup$ Thanks. Actually, I do not endow ML methods with "superior" powers, simply mention that some ML answers may be proposed for complex models. And by "true models", I mean the assessment of the performances of a method assuming (wrongly) that the data is produced by the said model. Which is a defect of most statistical analyses imo. $\endgroup$ – Xi'an May 13 '17 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.