Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

What is the reason that a likelihood function is not a pdf (probability density function)?

share|improve this question
4  
The likelihood function is a function of the unknown parameter $\theta$ (conditioned on the data). As such, it does typically not have area 1 (i.e. the integral over all possible values of $\theta$ is not 1) and is therefore by definition not a pdf. –  MånsT Jun 27 '12 at 18:35
3  
The same question on MO 2 years ago: mathoverflow.net/questions/10971/… –  Douglas Zare Jun 27 '12 at 20:35
2  
Interesting reference, @Douglas. The answers are rather unsatisfactory, IMHO. The accepted one assumes things that just aren't true ("both $p(X|m)$ and $p(m|X)$ are pdfs": not!) and the others don't really get at the statistical issues. –  whuber Jun 27 '12 at 22:18
1  
+1 whuber. This is amazing that there are so bad answers in the mathoverflow site in spite of its so high mathematical level ! –  Stéphane Laurent Jun 28 '12 at 18:28
1  
@Stephane: This is true, but statisticians and even probabilists seem to be fairly few and far between on MO, with some notable exceptions. That question was from fairly early in MO's existence when both the generally admissible questions and quality of answers were substantially different. –  cardinal Jun 28 '12 at 20:30

3 Answers 3

up vote 19 down vote accepted

We'll start with two definitions:

  • A probability density function (pdf) is a non-negative function that integrates to $1$.

  • The likelihood is defined as the joint density of the observed data as a function of the parameter. But, as pointed out by the reference to Lehmann made by @whuber in a comment below, the likelihood function is a function of the parameter only, with the data held as a fixed constant. So the fact that it is a density as a function of the data is irrelevant.

Therefore, the likelihood function is not a pdf because its integral with respect to the parameter does not necessarily equal 1 (and may not be integrable at all, actually, as pointed out by another comment from @whuber).

To see this, we'll use a simple example. Suppose you have a single observation, $x$, from a ${\rm Bernoulli}(\theta)$ distribution. Then the likelihood function is

$$ L(\theta) = \theta^{x} (1 - \theta)^{1-x} $$

It is a fact that $\int_{0}^{1} L(\theta) d \theta = 1/2$. Specifically, if $x = 1$, then $L(\theta) = \theta$, so $$\int_{0}^{1} L(\theta) d \theta = \int_{0}^{1} \theta \ d \theta = 1/2$$

and a similar calculation applies when $x = 0$. Therefore, $L(\theta)$ cannot be a density function.

Perhaps even more important than this technical example showing why the likelihood isn't a probability density is to point out that the likelihood is not the probability of the parameter value being correct or anything like that - it is the probability (density) of the data given the parameter value, which is a completely different thing. Therefore one should not expect the likelihood function to behave like a probability density.

share|improve this answer
6  
+1 A subtle point is that even the appearance of the "$d\theta$" in the integral is not part of the likelihood function; it comes from nowhere. Among the many ways to see this, consider that a reparameterization changes nothing essential about the likelihood--it is merely a renaming of the parameter--but will change the integral. E.g., if we parameterized the Bernoulli distributions with the log odds $\psi=\log(\theta/(1-\theta))$ then the integral would not even converge. –  whuber Jun 27 '12 at 21:03
    
@whuber - I know that MLEs are invariant to monotone transformation but clearly the shape of the likelihood as a function of $\psi$ is different from the shape as a function of $\theta$, so the integrals would be different. Isn't this akin to how the normalizing constant can be different when you transform a random variable? Perhaps I've missed your point.. –  Macro Jun 28 '12 at 0:48
1  
That's one way to put it: MLEs are invariant under monotone transformations but probability densities are not, QED! This was exactly Fisher's argument, which I have sketched in a comment to @Michael Chernick's reply. –  whuber Jun 28 '12 at 0:53
1  
+1 for whuber's comment. The "$d\theta$" has not even a sense in general because there's not even a $\sigma$-field in the parameter space ! –  Stéphane Laurent Jun 28 '12 at 18:15
1  
@PatrickCaldon The only continuity constraint is on the cdf, which requires right-continuity. You need this so your probability doesn't go from defined to undefined and (possibly) back again, which would be weird. I'm not 100% sure but I think so long as you have your cdf, and so a probability, you don't even have to be able to solve $\int_D f$. If you can that just ensures that the RV is continuous. –  Joey Jun 29 '12 at 19:58

I'm not a statistician, but my understanding is that while the likelihood function itself is not a PDF with respect to the parameter(s), it is directly related to that PDF by Bayes Rule. The likelihood function, P(X|theta), and posterior distribution, f(theta|X), are tightly linked; not "a completely different thing" at all.

share|improve this answer
1  
Welcome to our site! You might find interesting material in the comments to other answers in this thread. Some of them point out why Bayes' Rule does not apply unless additional mathematical machinery is explicitly introduced (such as a Sigma field for the parameter). –  whuber Feb 2 at 20:31
    
Thanks @whuber. I didn't notice any references to Bayes' Rule elsewhere in the thread, but I suppose there are allusions in the comments, assuming one is sufficiently fluent in graduate-level probability to pick up on them (which I'm not). Would you not agree that placing the likelihood function in the context of Bayes' Rule provides useful intuition for the OP's question? –  santayana Feb 2 at 21:09
    
Applying Bayes' rule is not possible without assuming a probability distribution for $\theta$: the distinction between that distribution, and the distribution of the data as a function of $\theta$, is what almost everything in this thread is about. Implicitly assuming there is, or can be, such a distribution is the source of the confusion discussed in the comment thread to Michael Chernick's answer. I would therefore agree that a clear and careful discussion of this point could be helpful, but anything short of that risks creating greater confusion. –  whuber Feb 2 at 22:02
    
My apologies, at first glance that thread seemed to amount to little more than a misunderstanding, but now I see the relevant comments you refer to, in particular your quote of Fisher. But does this not come down to a Bayesian v. Frequentist debate? Isn't there a large number of practitioners of Bayesian inference who would argue in favour of a probability distribution for theta? (whether you agree with them is another matter...) –  santayana Feb 2 at 22:44
    
Yes, the B vs. F debate is lurking here. A thoughtful frequentist will happily use Bayes' Rule when there exists a basis to adopt a prior distribution for $\theta$, but parts company from Bayesians by denying that we must adopt a prior. We can take our cue from how this question was phrased. If it had instead asked "why can one treat the likelihood function as a PDF (for the parameters)," that would have steered this conversation along Bayesian lines. But by asking it in the negative, the O.P. was looking for us to examine the likelihood from a frequentist point of view. –  whuber Feb 2 at 23:16

Okay but the likelihood function is the joint probability density for the observed data given the parameter $θ$. As such it can be normalized to form a probability density function. So it is essentially a pdf.

share|improve this answer
2  
So, you're just pointing out that the likelihood is integrable with respect to the parameter (is that always true?). I suppose you may be alluding to the likelihood's relationship to the posterior distribution when a flat prior is used, but without more explanation this answer remains mysterious to me. –  Macro Jun 27 '12 at 21:45
2  
Integrating to unity is beside the point. Fisher, in a 1922 paper On the Mathematical Foundations of Theoretical Statistics, observed that indeed usually the likelihood $L(\theta)$ can be "normalized" to integrate to unity upon multiplying by a suitable function $p(\theta)$ so that $\int L(\theta)p(\theta)d\theta=1$. What he objected to is the arbitrariness: there are many $p$ that work. "...the word probability is wrongly used in such a connection: probability is a ratio of frequencies, and about the frequencies of such values we can know nothing whatever." –  whuber Jun 28 '12 at 0:51
1  
@Néstor (and Michael) - it appears that whuber and I both interpreted this question as asking why the likelihood is not a density function, as a function of $\theta$ so it appears we are answering different questions. Of course the likelihood is the density function of the observations (given the parameter value) - that is how it's defined. –  Macro Jun 28 '12 at 1:55
1  
Michael, I think we interpreted it that way because the likelihood is a function of $\theta$ so, if it were a density, then it would be a density in $\theta$. I can imagine interpreting it the way you have but that possibility didn't occur to me until after reading Nestor's comment. –  Macro Jun 28 '12 at 2:14
2  
I find the ambiguity is created by this answer but is not present in the question. As @Macro points out, the likelihood is a function only of the parameter. (E.g., "The density $f(x_1,\theta)\cdots f(x_n,\theta)$, considered for fixed $x$ as a function of $\theta$, is called the likelihood function: E. L. Lehmann, Theory of Point Estimation, section 6.2.) Thus the question is clear. Replying, then, that the "likelihood is the joint probability density" does not clarify but confuses the issue. –  whuber Jun 28 '12 at 13:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.