14
$\begingroup$

This question is spurred from the question: When (if ever) is a frequentist approach substantively better than a Bayesian?

As I posted in my solution to that question, in my opinion, if you are a frequentist you do not have to believe/adhere to the likelihood principle since often time frequentists methods will violate it. However, and this is usually under the assumption of proper priors, Bayesian methods never violate the likelihood principle.

So now, to say you are a Bayesian does that confirm one's belief or agreement in the likelihood principle, or is the argument that being a Bayesian just has the nice consequence that the likelihood principle does not get violated?

$\endgroup$
  • 4
    $\begingroup$ No - see the Jeffreys prior. Bayesian methods may violate the (strong) likelihood principle. $\endgroup$ – Scortchi Feb 7 '16 at 16:56
  • 6
    $\begingroup$ Yes indeed, Jeffreys priors and also solutions that use the data several times like posterior predictives are in violation of the likelihood principle but can still be deemed to be Bayesian... $\endgroup$ – Xi'an Feb 7 '16 at 17:07
  • 1
    $\begingroup$ Not necessarily. And I'm not sure what difference it makes. $\endgroup$ – Scortchi Feb 7 '16 at 17:15
  • 2
    $\begingroup$ Compare those for the binomial & negative binomial. $\endgroup$ – Scortchi Feb 7 '16 at 17:20
  • 1
    $\begingroup$ xianblog.wordpress.com/2014/11/13/… $\endgroup$ – Scortchi Feb 10 '16 at 13:51
13
$\begingroup$

In the use of Bayes' Theorem to calculate the posterior probabilities that constitute inference about model parameters, the weak likelihood principle is automatically adhered to:

$$\mathrm{posterior} \propto \mathrm{prior} \times \mathrm{likelihood}$$

Nevertheless, in some objective Bayesian approaches the sampling scheme determines the choice of prior, the motivation being that an uninformative prior should maximize the divergence between the prior and posterior distributions—letting the data have as much influence as possible. Thus they violate the strong likelihood principle.

Jeffreys priors, for instance, are proportional to the square root of the determinant of the Fisher information, an expectation over the sample space. Consider inference about the probability parameter $\pi$ of Bernoulli trials under binomial & negative binomial sampling. The Jeffreys priors are

$$ \def\Pr{\mathop{\rm Pr}\nolimits} \begin{align} \Pr_\mathrm{NB}(\pi) &\propto \pi^{-1} (1-\pi)^{-\tfrac{1}{2}}\\ \Pr_\mathrm{Bin}(\pi) &\propto \pi^{-\tfrac{1}{2}} (1-\pi)^{-\tfrac{1}{2}} \end{align} $$

& conditioning on $x$ successes from $n$ trials leads to the posterior distributions

$$ \begin{align} \Pr_\mathrm{NB}(\pi \mid x,n) \sim \mathrm{Beta}(x, n-x+\tfrac{1}{2})\\ \Pr_\mathrm{Bin}(\pi \mid x,n)\sim \mathrm{Beta}(x+\tfrac{1}{2}, n-x+\tfrac{1}{2}) \end{align} $$

So observing say 1 success from 10 trials would lead to quite different posterior distributions under the two sampling schemes:

enter image description here

Though following such rules for deriving uninformative priors can sometimes leave you with improper priors, that in itself isn't the root of the violation of the likelhood principle entailed by the practice. An approximation to the Jeffreys prior, $ \pi^{-1+c} (1-\pi)^{-1/2}$, where $0 < c\ll 1$, is quite proper, & makes negligible difference to the posterior.

You might also consider model checking—or doing anything as a result of your checks—as contrary to the weak likelihood principle; a flagrant case of using the ancillary part of the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.