11
$\begingroup$

I have sometimes seen some statisticians used bayesian analysis and related techniques such as MCMC simply as a tool when a frequentist approach is not satisfying, typically for example when the maximum likelihood estimator is hard to find or takes too much time to compute.

In these case they focus only on the definition of the model and estimation (by MCMC for example) and barely on the choice prior distributions (from what I've seen). I guess they use flat distributions by default but the particular choice of a prior is never discussed, maybe because of the large number of parameters the model may have and the difficulty to assign a meaningful prior distribution to each one of them.

Is it something usual to switch to a bayesian analysis and use it merely a as computational counterpart of a frequentist approach when the latter performs poorly?
My feeling is that bayesian and frequentist are very distinct statistical framework (at least that's what my teachers taught me!) and using one or another shouldn't be justified simply by computational purposes and moreover the choice of the prior distributions should at least be carefully made.

I know this is not really a statistical question but I just would like to know what pure bayesians think about this kind of use of bayesian analysis. Sorry in advance if it does not fit this site.

$\endgroup$
3
  • 1
    $\begingroup$ If in the frequentist case you cannot compute the likelihood then also in the bayesian case you will not be able to compute the posterior as it is the product of likelihood with prior. Also, I assume in the frequentist case you use some kind of optimization procedure which I believe in general is a faster procedure than running an MCMC algorithm. $\endgroup$
    – Fiodor1234
    Sep 25 at 19:24
  • 2
    $\begingroup$ A case that I can think of that a Bayesian analysis might be preferred to a frequentist, is when the parameter is not identifiable $\endgroup$
    – Fiodor1234
    Sep 25 at 19:25
  • $\begingroup$ "I have sometimes seen some statisticians used bayesian analysis and related techniques such as MCMC simply as a tool when a frequentist approach is not satisfying" Everything is a tool. Frequentist or Bayesian, there is no distinction. If some of these tools might be easier to compute then it certainly will be used more often (I am sure this works in both ways). $\endgroup$ Sep 27 at 14:05
10
$\begingroup$

The set of methods called "frequentist" statistics is quite broad. It allows you to specify any proposed estimator you want and then investigate its long-run properties conditional on the true values of the parameters. This method only counts an estimator out completely if it is "inadmissible", meaning that it is dominated by another available estimator (i.e., it gives equal/higher risk over every possible value of the parameter and higher risk over at least some parameter values).

Now, there is a famous theorem that says that, under wide conditions, Bayesian estimators are admissible --- i.e., they are not dominated by other estimators. Bayesian estimators tend to be biased (since they incorporate prior information) but they are also consistent under fairly wide conditions. This means that they are estimators that will tend to perform well in terms of the frequentist criteria. Consequently, frequentists usually consider these estimators as one option that can be used in their analysis.

By definition, "pure Bayesians" are going to adopt the Bayesian methodology in all cases. Most pure Bayesians are going to have adopted this methodology by being convinced of its underlying philosophical and mathematical superiority. However, part of the motivation for adoption of Bayesian methods may be the knowledge that even under the frequentist paradigm, these methods tend to perform well according to frequentist criteria. As to what a pure Bayesian would think of a frequentist using a Bayesian estimator, I suppose it is somewhat like what a priest would think of an atheist who decides one day to pray for spiritual guidance (e.g., on the basis that it can't do any harm even under their own philosophy). They would likely see this as a desirable change in behaviour, improperly motivated, but also possibly a useful entry-point to try to convince them that the general philosophy underpinning that activity is coherent and desirable.

$\endgroup$
3
  • 3
    $\begingroup$ This makes the matter more tribal or theological than I think it now is. Here is a naive example. Is looking at a graph of data alone "frequentist" and so should "pure Bayesians" (your term not mine) decline to do it? I think many statistical people would recognise that the Bayesians have excellent arguments but don't practise them because (a) they were not trained that way (b) it is, or appears to be, hard work to re-educate yourself (c) the field you are in expects mostly frequentist methods. These are mundane practicalities, but often personally compelling. $\endgroup$
    – Nick Cox
    Sep 26 at 9:10
  • $\begingroup$ @NickCox: No, not my term; OP's term. I agree that OP is asking about a hypothetical extreme class of people, but that is what he is asking. (I also put "pure Bayesians" in quotes because I share your scepticism of the term.) $\endgroup$
    – Ben
    Sep 26 at 11:07
  • 2
    $\begingroup$ Sorry about that. My fault in missing the original mention. This is a case where the question is familiar and I am interested in the answers.... $\endgroup$
    – Nick Cox
    Sep 26 at 11:22
8
$\begingroup$

In a 2002 paper with Arnaud Doucet and Simon Godsill, Marginal maximum a posteriori estimation using Markov chain Monte Carlo, we use an MCMC approach to derive the maximum likelihood estimator in latent variable models where the observed likelihood is not available. By repeating the number of repetitions of said latent variables in a simulated annealing spirit. Similar proposals were proposed subsequently by

$\endgroup$
6
$\begingroup$

Comment: Here are a few reasons why a frequentist statistician might use a Bayesian approach.

Computational convenience, as @Fiodor1234 says, may not be high on the list. One such example might be use of the Jeffreys posterior probability interval as a confidence interval for a binomial proportion. For example, if you have $x = 42$ successes in $n=100$ trials, the asymptotic Wald interval is not the best choice because of the small sample size. The Agresti-Coull interval is easy to compute and comes close to more accurate intervals that are somewhat intricate to compute. The Jeffreys interval, based on the noninformative Bayesian prior $\mathsf{Beta}(.5, .5)$, is easy to compute in R and has good frequentist properties.

p.hat = 42/100
CI.Wald = p.hat + qnorm(c(.025,.975))*sqrt(p.hat*(1-p.hat)/100)
round(CI.Wald,4)
[1] 0.3233 0.5167

p.est = (42+2)/(100+4)
CI.Agr = p.est + qnorm(c(.025,.975))*sqrt(p.est*(1-p.est)/104)
round(CI.Agr,4)
[1] 0.3281 0.5180

CI.Jeff = qbeta(c(.025,.975), 42+.5, 100-42+.5)
round(CI.Jeff,4)
[1] 0.3267 0.5179

Proper support of distribution. In an attempt to find the prevalence of a disease from screening test data, traditional methods can give an interval for prevalence that extends beyond $(0,1).$ By using a Gibbs sampler with a beta prior distribution, it is possible to get a useful interval estimate for prevalence. (Since the beta prior has the unit interval as support, then the posterior distribution will also.) See example..

'Simulate' latent data. Sometimes one wants to test or to give a parameter estimate for latent data, which can be reliably reconstructed using a Gibbs sampler. One simple example is to know the variability of groups in a one-way random-effects ANOVA. Observed values from the groups are available, but the components of variance due to the various groups (separate from overall variance) are typically latent.

$\endgroup$
3
  • $\begingroup$ Edits show 4 decimal places for CIs and repair typos. $\endgroup$
    – BruceET
    Sep 26 at 1:09
  • $\begingroup$ “[H]as good frequentist properties” is a consequence of the Jeffreys interval being computed by a conjugate prior? $\endgroup$ Sep 27 at 13:19
  • 1
    $\begingroup$ @SingleMalt. See Wikipedia under Jeffreys Interval. Beta prior conjugate to binomial likelihood gives relevant beta posterior. // Suspect that claim there of good frequentist properties is consensus based on fairly widespread use. // Prior $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(1,1)$ is sometimes used instead of Jeffreys prior $\mathsf{Beta}(.5,.5)$ preferred by Bayesians for technical reasons. $\endgroup$
    – BruceET
    Sep 27 at 17:54
3
$\begingroup$

Other reasons to use Bayesian approaches include

  • getting more accurate inference when the log-likelihood is very non-Gaussian. For example, in binary logistic regression standard p-values and confidence intervals may be inaccurate whereas Bayesian quantities are exact.
  • getting accurate uncertainty intervals for complex derived parameters. In the frequentist world we frequently have to resort to the delta method to get approximate confidence intervals. Not only is this labor intensive, but the result is often very unsatisfactory because such intervals are forced to be symmetric when they should have been asymmetric in order to have accurate coverage with respect to both tails. One example is state occupancy probabilities in a state transition model, which involve recursive matrix multiplications are a mess to deal with in the frequentist domain. With MCMC (let's say you have 4000 posterior draws from the multivariate distribution of all parameters together) you just compute the complex derived parameter 4000 times and estimate the highest posterior density interval from those 4000 numbers.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.