17
$\begingroup$

I currently try to understand Likelihood Principle and I frankly don't get it at all. So, I will write all my question as a list, even if those might be pretty basic questions.

  • What exactly does "all of the information" phrase mean in the context of this principle? (as in all of the information in a sample is contained in the likelihood function.)
  • Is the principle somehow connected to the very provable fact, that $p(x|y)\propto p(y|x)p(x)$? Is the "likelihood" in the principle the same thing, as $p(y|x)$, or not?
  • How can a mathematical theorem be "controversial"? My (weak) understanding of math is that a theorem is either proven, or is not proven. To what category does Likelihood Principle fall?
  • How is the Likelihood Principle important for Bayesian inference, which is based on $p(x|y)\propto p(y|x)p(x)$ formula?
Improve this question
$\endgroup$
2

1 Answer 1

Reset to default
12
$\begingroup$

The likelihood principle has been stated in many different ways, with variable meaning and intelligibility. A.W.F. Edwards's book Likelihood is both an excellent introduction to many aspects of likelihood and still in print. This is how Edwards defines the likelihood principle:

"Within the framework of a statistical model, all of the information which the data provide concerning the relative merits of two hypotheses is contained in the likelihood ratio of those hypotheses." (Edwards 1972, 1992 p. 30)

So now to answers.

  1. "All of the information in the sample", as you quote, is simply an inadequate expression of the relevant part of the likelihood principle. Edwards says it much better: the model matters and the relevant information is the information relating to the relative merits of hypotheses. It is useful to note that the likelihood ratio only makes sense where the hypotheses in question come from the same statistical model and are mutually exclusive. In effect, they have to be points on the same likelihood function for the ratio to be useful.

  2. The likelihood principle is related to Bayes theorem, as you can see, but it is provable without reference to Bayes theorem. Yes, p(x|y) is (proportional to) a likelihood as long as x is data and y is a hypothesis (which might just be a hypothesised parameter value).

  3. The likelihood principle is controversial because its proof has been contested. In my opinion the disproofs are faulty, but nonetheless it is controversial. (At a different level, it can be said that the likelihood principle is controversial because it implies that frequentist methods for inference are in some ways faulty. Some people don't like that.) The likelihood principle has been proved, but its scope of relevance may be more constrained than its critics imagine.

  4. The likelihood principle is important for Bayesian methods because the data enter into Bayes equation by way of the likelihoods. Most Bayesian methods are compliant with the likelihood principle, but not all. Some people, like Edwards and Royall, contend that inferences can be made on the basis of likelihood functions without use of Bayes theorem, "pure likelihood inference". That is controversial as well. In fact, it is probably more controversial than the likelihood principle because Bayesians tend to agree with frequentists that pure likelihood methods are inappropriate. (My enemy's enemy...)

Improve this answer
$\endgroup$
7
  • $\begingroup$ "It is useful to note that the likelihood ratio only makes sense where the hypotheses in question come from the same statistical model" - what does that mean exactly? It sounds like you're saying you can't compare models from different families of distributions, which ain't so. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jul 25, 2013 at 8:14
  • $\begingroup$ Because the likelihoods are only proportional to *p*(x|y) there is always an unknown proportionality constant. Different statistical models allow different proportionality constants and so the likelihoods can be incommensurable. $\endgroup$
    – Michael Lew
    Jul 25, 2013 at 20:42
  • $\begingroup$ Sometimes different models can be arranged to yield a single likelihood function (often multidimensional) so that the likelihoods can sensibly be compared, but that is not always possible. $\endgroup$
    – Michael Lew
    Jul 25, 2013 at 20:44
  • 1
    $\begingroup$ I'm perhaps missing some subtlety, but my understanding is that there are no unknown constants in a likelihood, just constants that cancel when you calculate the likelihood ratio for models from the same family, & so get ignored then. Anyway: for data $\vec{x}$ & any densities $f$ & $g$ with parameters $\theta$ & $\phi$, I'd call the statistic $$\frac{f\left(\vec{x};\hat{\theta}\right)}{g\left(\vec{x};\hat{\phi}\right)}$$ a likelihood ratio; & it can be used for inference. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jul 26, 2013 at 10:00
  • 1
    $\begingroup$ See Cox (1961), Tests of Separate Families of Hypotheses, Proc. 4th Berkeley Symp. on Math. Statist. and Prob. 1. Of course Wilks' theorem doesn't apply, so twice its logarithm is not distributed as $\chi^2$. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jul 26, 2013 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.