Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 11 down vote favorite
2

What is the reason that a likelihood function is not a pdf (probability density function)?

share|improve this question
4  
The likelihood function is a function of the unknown parameter $\theta$ (conditioned on the data). As such, it does typically not have area 1 (i.e. the integral over all possible values of $\theta$ is not 1) and is therefore by definition not a pdf. – MånsT Jun 27 '12 at 18:35
3  
The same question on MO 2 years ago: mathoverflow.net/questions/10971/… – Douglas Zare Jun 27 '12 at 20:35
2  
Interesting reference, @Douglas. The answers are rather unsatisfactory, IMHO. The accepted one assumes things that just aren't true ("both $p(X|m)$ and $p(m|X)$ are pdfs": not!) and the others don't really get at the statistical issues. – whuber♦ Jun 27 '12 at 22:18
1  
+1 whuber. This is amazing that there are so bad answers in the mathoverflow site in spite of its so high mathematical level ! – Stéphane Laurent Jun 28 '12 at 18:28
1  
@Stephane: This is true, but statisticians and even probabilists seem to be fairly few and far between on MO, with some notable exceptions. That question was from fairly early in MO's existence when both the generally admissible questions and quality of answers were substantially different. – cardinal Jun 28 '12 at 20:30
show 1 more comment

2 Answers

up vote 15 down vote

We'll start with two definitions:

  • A probability density function (pdf) is a non-negative function that integrates to $1$.

  • The likelihood is defined as the joint density of the observed data as a function of the parameter. But, as pointed out by the reference to Lehmann made by @whuber in a comment below, the likelihood function is a function of the parameter only, with the data held as a fixed constant. So the fact that it is a density as a function of the data is irrelevant.

Therefore, the likelihood function is not a pdf because its integral with respect to the parameter does not necessarily equal 1 (and may not be integrable at all, actually, as pointed out by another comment from @whuber).

To see this, we'll use a simple example. Suppose you have a single observation, $x$, from a ${\rm Bernoulli}(\theta)$ distribution. Then the likelihood function is

$$ L(\theta) = \theta^{x} (1 - \theta)^{1-x} $$

It is a fact that $\int_{0}^{1} L(\theta) d \theta = 1/2$. Specifically, if $x = 1$, then $L(\theta) = \theta$, so $$\int_{0}^{1} L(\theta) d \theta = \int_{0}^{1} \theta \ d \theta = 1/2$$

and a similar calculation applies when $x = 0$. Therefore, $L(\theta)$ cannot be a density function.

Perhaps even more important than this technical example showing why the likelihood isn't a probability density is to point out that the likelihood is not the probability of the parameter value being correct or anything like that - it is the probability (density) of the data given the parameter value, which is a completely different thing. Therefore one should not expect the likelihood function to behave like a probability density.

share|improve this answer
5  
+1 A subtle point is that even the appearance of the "$d\theta$" in the integral is not part of the likelihood function; it comes from nowhere. Among the many ways to see this, consider that a reparameterization changes nothing essential about the likelihood--it is merely a renaming of the parameter--but will change the integral. E.g., if we parameterized the Bernoulli distributions with the log odds $\psi=\log(\theta/(1-\theta))$ then the integral would not even converge. – whuber♦ Jun 27 '12 at 21:03
@whuber - I know that MLEs are invariant to monotone transformation but clearly the shape of the likelihood as a function of $\psi$ is different from the shape as a function of $\theta$, so the integrals would be different. Isn't this akin to how the normalizing constant can be different when you transform a random variable? Perhaps I've missed your point.. – Macro Jun 28 '12 at 0:48
That's one way to put it: MLEs are invariant under monotone transformations but probability densities are not, QED! This was exactly Fisher's argument, which I have sketched in a comment to @Michael Chernick's reply. – whuber♦ Jun 28 '12 at 0:53
A quibble - is that the right definition of PDF? Suppose we have $f(x) = 1$ if $x$ is between 0 and 1 and irrational, and $f(x) = 0$ otherwise. My probability/measure theory is too rusty to remember if you need continuity or something stronger/weaker, but I can see my $f$ giving some very anti-intuitive results. – Patrick Caldon Jun 28 '12 at 1:56
@PatrickCaldon - I think any non-negative integrable function is proportional to a probability density. You surely do not need continuity - consider the density of a variable defined by a mixture distribution e.g. a 50/50 mixture between a ${\rm uniform}(0,1)$ and ${\rm uniform}(2,3)$ has a discontinuous density. I guess I don't see the problem with the example you've given. If you can calculate $ \int_{D} f$ then you can assign a probability to the set $D$. – Macro Jun 28 '12 at 2:10
show 7 more comments
up vote 0 down vote

Okay but the likelihood function is the joint probability density for the observed data given the parameter $θ$. As such it can be normalized to form a probability density function. So it is essentially a pdf.

share|improve this answer
2  
So, you're just pointing out that the likelihood is integrable with respect to the parameter (is that always true?). I suppose you may be alluding to the likelihood's relationship to the posterior distribution when a flat prior is used, but without more explanation this answer remains mysterious to me. – Macro Jun 27 '12 at 21:45
2  
Integrating to unity is beside the point. Fisher, in a 1922 paper On the Mathematical Foundations of Theoretical Statistics, observed that indeed usually the likelihood $L(\theta)$ can be "normalized" to integrate to unity upon multiplying by a suitable function $p(\theta)$ so that $\int L(\theta)p(\theta)d\theta=1$. What he objected to is the arbitrariness: there are many $p$ that work. "...the word probability is wrongly used in such a connection: probability is a ratio of frequencies, and about the frequencies of such values we can know nothing whatever." – whuber♦ Jun 28 '12 at 0:51
1  
@Néstor (and Michael) - it appears that whuber and I both interpreted this question as asking why the likelihood is not a density function, as a function of $\theta$ so it appears we are answering different questions. Of course the likelihood is the density function of the observations (given the parameter value) - that is how it's defined. – Macro Jun 28 '12 at 1:55
1  
Michael, I think we interpreted it that way because the likelihood is a function of $\theta$ so, if it were a density, then it would be a density in $\theta$. I can imagine interpreting it the way you have but that possibility didn't occur to me until after reading Nestor's comment. – Macro Jun 28 '12 at 2:14
2  
I find the ambiguity is created by this answer but is not present in the question. As @Macro points out, the likelihood is a function only of the parameter. (E.g., "The density $f(x_1,\theta)\cdots f(x_n,\theta)$, considered for fixed $x$ as a function of $\theta$, is called the likelihood function: E. L. Lehmann, Theory of Point Estimation, section 6.2.) Thus the question is clear. Replying, then, that the "likelihood is the joint probability density" does not clarify but confuses the issue. – whuber♦ Jun 28 '12 at 13:10
show 8 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.