89
$\begingroup$

What is the reason that a likelihood function is not a pdf (probability density function)?

$\endgroup$
6
  • 15
    $\begingroup$ The likelihood function is a function of the unknown parameter $\theta$ (conditioned on the data). As such, it does typically not have area 1 (i.e. the integral over all possible values of $\theta$ is not 1) and is therefore by definition not a pdf. $\endgroup$
    – MånsT
    Jun 27, 2012 at 18:35
  • 6
    $\begingroup$ The same question on MO 2 years ago: mathoverflow.net/questions/10971/… $\endgroup$ Jun 27, 2012 at 20:35
  • 6
    $\begingroup$ Interesting reference, @Douglas. The answers are rather unsatisfactory, IMHO. The accepted one assumes things that just aren't true ("both $p(X|m)$ and $p(m|X)$ are pdfs": not!) and the others don't really get at the statistical issues. $\endgroup$
    – whuber
    Jun 27, 2012 at 22:18
  • 4
    $\begingroup$ +1 whuber. This is amazing that there are so bad answers in the mathoverflow site in spite of its so high mathematical level ! $\endgroup$ Jun 28, 2012 at 18:28
  • 3
    $\begingroup$ @Stephane: This is true, but statisticians and even probabilists seem to be fairly few and far between on MO, with some notable exceptions. That question was from fairly early in MO's existence when both the generally admissible questions and quality of answers were substantially different. $\endgroup$
    – cardinal
    Jun 28, 2012 at 20:30

5 Answers 5

88
$\begingroup$

We'll start with two definitions:

  • A probability density function (pdf) is a non-negative function that integrates to $1$.

  • The likelihood is defined as the joint density of the observed data as a function of the parameter. But, as pointed out by the reference to Lehmann made by @whuber in a comment below, the likelihood function is a function of the parameter only, with the data held as a fixed constant. So the fact that it is a density as a function of the data is irrelevant.

Therefore, the likelihood function is not a pdf because its integral with respect to the parameter does not necessarily equal 1 (and may not be integrable at all, actually, as pointed out by another comment from @whuber).

To see this, we'll use a simple example. Suppose you have a single observation, $x$, from a ${\rm Bernoulli}(\theta)$ distribution. Then the likelihood function is

$$ L(\theta) = \theta^{x} (1 - \theta)^{1-x} $$

It is a fact that $\int_{0}^{1} L(\theta) d \theta = 1/2$. Specifically, if $x = 1$, then $L(\theta) = \theta$, so $$\int_{0}^{1} L(\theta) d \theta = \int_{0}^{1} \theta \ d \theta = 1/2$$

and a similar calculation applies when $x = 0$. Therefore, $L(\theta)$ cannot be a density function.

Perhaps even more important than this technical example showing why the likelihood isn't a probability density is to point out that the likelihood is not the probability of the parameter value being correct or anything like that - it is the probability (density) of the data given the parameter value, which is a completely different thing. Therefore one should not expect the likelihood function to behave like a probability density.

$\endgroup$
12
  • 23
    $\begingroup$ +1 A subtle point is that even the appearance of the "$d\theta$" in the integral is not part of the likelihood function; it comes from nowhere. Among the many ways to see this, consider that a reparameterization changes nothing essential about the likelihood--it is merely a renaming of the parameter--but will change the integral. E.g., if we parameterized the Bernoulli distributions with the log odds $\psi=\log(\theta/(1-\theta))$ then the integral would not even converge. $\endgroup$
    – whuber
    Jun 27, 2012 at 21:03
  • 5
    $\begingroup$ That's one way to put it: MLEs are invariant under monotone transformations but probability densities are not, QED! This was exactly Fisher's argument, which I have sketched in a comment to @Michael Chernick's reply. $\endgroup$
    – whuber
    Jun 28, 2012 at 0:53
  • 10
    $\begingroup$ +1 for whuber's comment. The "$d\theta$" has not even a sense in general because there's not even a $\sigma$-field in the parameter space ! $\endgroup$ Jun 28, 2012 at 18:15
  • 1
    $\begingroup$ @PatrickCaldon The only continuity constraint is on the cdf, which requires right-continuity. You need this so your probability doesn't go from defined to undefined and (possibly) back again, which would be weird. I'm not 100% sure but I think so long as you have your cdf, and so a probability, you don't even have to be able to solve $\int_D f$. If you can that just ensures that the RV is continuous. $\endgroup$
    – Joey
    Jun 29, 2012 at 19:58
  • 1
    $\begingroup$ @hbak It's the joint density of the sample data. It only becomes a product if we further assume that the samples are independent of one another. $\endgroup$
    – SigmaX
    Dec 5, 2018 at 15:16
6
$\begingroup$

Okay but the likelihood function is the joint probability density for the observed data given the parameter $θ$. As such it can be normalized to form a probability density function. So it is essentially like a pdf.

$\endgroup$
13
  • 4
    $\begingroup$ So, you're just pointing out that the likelihood is integrable with respect to the parameter (is that always true?). I suppose you may be alluding to the likelihood's relationship to the posterior distribution when a flat prior is used, but without more explanation this answer remains mysterious to me. $\endgroup$
    – Macro
    Jun 27, 2012 at 21:45
  • 8
    $\begingroup$ Integrating to unity is beside the point. Fisher, in a 1922 paper On the Mathematical Foundations of Theoretical Statistics, observed that indeed usually the likelihood $L(\theta)$ can be "normalized" to integrate to unity upon multiplying by a suitable function $p(\theta)$ so that $\int L(\theta)p(\theta)d\theta=1$. What he objected to is the arbitrariness: there are many $p$ that work. "...the word probability is wrongly used in such a connection: probability is a ratio of frequencies, and about the frequencies of such values we can know nothing whatever." $\endgroup$
    – whuber
    Jun 28, 2012 at 0:51
  • 3
    $\begingroup$ @Néstor (and Michael) - it appears that whuber and I both interpreted this question as asking why the likelihood is not a density function, as a function of $\theta$ so it appears we are answering different questions. Of course the likelihood is the density function of the observations (given the parameter value) - that is how it's defined. $\endgroup$
    – Macro
    Jun 28, 2012 at 1:55
  • 5
    $\begingroup$ I find the ambiguity is created by this answer but is not present in the question. As @Macro points out, the likelihood is a function only of the parameter. (E.g., "The density $f(x_1,\theta)\cdots f(x_n,\theta)$, considered for fixed $x$ as a function of $\theta$, is called the likelihood function: E. L. Lehmann, Theory of Point Estimation, section 6.2.) Thus the question is clear. Replying, then, that the "likelihood is the joint probability density" does not clarify but confuses the issue. $\endgroup$
    – whuber
    Jun 28, 2012 at 13:10
  • 3
    $\begingroup$ @whuber I guess technically the common usage of the term likelihood function means what Lehmann defined it as in his book. But the function itself can be viewed as a function of the parameters for fixed values of the observations or as a function of the observations for given values of the parameters. So when the question is asked "Why is the likelihood function not a pdf?" I was think that one can view it as a pdf for the data given the parameters. I think it is a plausible interpretation. But I can now see how you would interpret it your way based on the formal definition. $\endgroup$ Jun 28, 2012 at 14:06
3
$\begingroup$

The likelihood is defined as $\mathcal{L}(\theta; x_1,...,x_n) = f(x_1,...,x_n; \theta)$, where if f(x; θ) is a probability mass function, then the likelihood is always less than one, but if f(x; θ) is a probability density function, then the likelihood can be greater than one, since densities can be greater than one.

Normally observations are treated iid, then:
$\mathcal{L}(\theta; x_1,...,x_n) = f(x_1,...,x_n; \theta) = \prod_{j} f(x_j; \theta)$

Let's see its original form:

According to the Bayesian inference, $f(x_1,...,x_n; \theta) = \frac{f(\theta; x_1,...,x_n) * f(x_1,...,x_n)}{f(\theta)}$ holds, that is $\hat{\mathcal{L}} = \frac{posterior * evidence}{prior}$. Notice that the maximum likelihood estimate treats the ratio of evidence to prior as a constant(see answers of this question), which omits the prior beliefs. The likelihood has a positive correlation with the posterior which is based on the estimated parameters. $\hat{\mathcal{L}}$ may be a pdf but $\mathcal{L}$ is not since $\mathcal{L}$ is just a part of $\hat{\mathcal{L}}$ which is intractable.

For example, I don't know the mean and standard variance of a Gaussian distribution and want to get them by training using a lot of observation from that distribution. I first initialize the mean and standard variance randomly(which defines a Gaussian distribution), and then I take one case and fit into the estimated distribution and I can get a probability from the estimated distribution. Then I continue to put the case in and get many probabilities and then I multiply these probabilities and get a score. This kind of score is the likelihood. Hardly can it be a probability of a certain pdf.

$\endgroup$
0
$\begingroup$

I'm not a statistician, but my understanding is that while the likelihood function itself is not a PDF with respect to the parameter(s), it is directly related to that PDF by Bayes Rule. The likelihood function, P(X|theta), and posterior distribution, f(theta|X), are tightly linked; not "a completely different thing" at all.

$\endgroup$
5
  • 3
    $\begingroup$ Welcome to our site! You might find interesting material in the comments to other answers in this thread. Some of them point out why Bayes' Rule does not apply unless additional mathematical machinery is explicitly introduced (such as a Sigma field for the parameter). $\endgroup$
    – whuber
    Feb 2, 2015 at 20:31
  • 1
    $\begingroup$ Thanks @whuber. I didn't notice any references to Bayes' Rule elsewhere in the thread, but I suppose there are allusions in the comments, assuming one is sufficiently fluent in graduate-level probability to pick up on them (which I'm not). Would you not agree that placing the likelihood function in the context of Bayes' Rule provides useful intuition for the OP's question? $\endgroup$
    – santayana
    Feb 2, 2015 at 21:09
  • 1
    $\begingroup$ Applying Bayes' rule is not possible without assuming a probability distribution for $\theta$: the distinction between that distribution, and the distribution of the data as a function of $\theta$, is what almost everything in this thread is about. Implicitly assuming there is, or can be, such a distribution is the source of the confusion discussed in the comment thread to Michael Chernick's answer. I would therefore agree that a clear and careful discussion of this point could be helpful, but anything short of that risks creating greater confusion. $\endgroup$
    – whuber
    Feb 2, 2015 at 22:02
  • $\begingroup$ My apologies, at first glance that thread seemed to amount to little more than a misunderstanding, but now I see the relevant comments you refer to, in particular your quote of Fisher. But does this not come down to a Bayesian v. Frequentist debate? Isn't there a large number of practitioners of Bayesian inference who would argue in favour of a probability distribution for theta? (whether you agree with them is another matter...) $\endgroup$
    – santayana
    Feb 2, 2015 at 22:44
  • 3
    $\begingroup$ Yes, the B vs. F debate is lurking here. A thoughtful frequentist will happily use Bayes' Rule when there exists a basis to adopt a prior distribution for $\theta$, but parts company from Bayesians by denying that we must adopt a prior. We can take our cue from how this question was phrased. If it had instead asked "why can one treat the likelihood function as a PDF (for the parameters)," that would have steered this conversation along Bayesian lines. But by asking it in the negative, the O.P. was looking for us to examine the likelihood from a frequentist point of view. $\endgroup$
    – whuber
    Feb 2, 2015 at 23:16
-1
$\begingroup$

yoooo, lets make something clear. Likelihood is completely different from probability!, when we want to calculate the probability of for example getting x=0, when x is coming from a normal distribution with miu=0 and sigma=1, we need to define a bin, like 0.01, and integral the probability function there(pdf, in this case normal distribution). so we calculate the integral of normal distribution and put, for instance, -0.01 and 0.01 as input for outcome of integral.

BUTT, in likelihood, we just calculate the point value of the pdf... , this is totally different from the probability of the x to be 0, we just enter input x=0 to the function and get the outcome. for instance, by definition, the function y=1, for x=(0 to 1), the integral of this function is 1, so this can be a pdf, but the point value of each x in (0 to 1) is equal 1, which is not the probability of this ponits, just the value of pdf in those point.

now we get to why likelihood is used in modeling, when we maximize the likelihood function for a set of observed data, with respect to a assumed model(function with parameters to be found), in action this would maximize the probability of those data with respect to that model(function). we work with the likelihood, but in the end, probability (integral of pdf) become maximum as well.

edit1: for the second comment, youre right, my bad, I meant the integral of pdf over the data that we collected. the whole purpose of modeling is to fit a 'decided' model to a set of data in a way that the probability of those data considering our 'decided' model would be maximized. for this probability to be maximized, we need to calculate the integral of pdf over our data. by maximizing likelihood, we maximize the integral as well.

$\endgroup$
4
  • 3
    $\begingroup$ What does this add to already existing answers? $\endgroup$ Dec 26, 2021 at 11:58
  • $\begingroup$ Because the "integral of [the] pdf" is, by definition, always $1,$ it is hard to see what your last point might be. $\endgroup$
    – whuber
    Dec 26, 2021 at 14:02
  • $\begingroup$ Re the edit: because data are always finite, what do you mean by "integral of pdf over the data"?? $\endgroup$
    – whuber
    Dec 27, 2021 at 15:08
  • $\begingroup$ exactly, what i mentioned in text, for instance, the probability of getting x=0 from normal distribution with miu=0 and sigma=1, is '0'. for every point the probability is '0'. in general, the probability of finding someone with the height exactly equal to 180cm is '0', when i say 180cm, i mean 180.000000000000 ... cm, but if we define a bin, then we can calculate the probability of that "event" over our pdf. for this instance, we can calculate the integral of pdf over 179.5 and 180.5 cm, in order to calculate the probability of selecting someone randomly with the height 180 from a population. $\endgroup$ Dec 28, 2021 at 15:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.