1
$\begingroup$

Before anyone says this has been answered elsewhere I don't think it has.

The likelihood is given by:

$$ L(θ;x_1,\cdots, x_n) = \prod^n_i f(x_i\mid\theta)$$

where $f$ is the probability density function for a continuous function or probability mass function if discrete.

For a discrete probability distribution, the individual $f$ terms are probabilities so I understand the need to form a product.

For a continuous probability distribution such as the normal distribution, the probability is the area, not the height. Why can we multiply the heights for a pdf when computing the likelihood since the height itself isn't a probability?

$\endgroup$
20
  • $\begingroup$ @whuber : I think you're quite mistaken in calling this a duplicate. Look closely at the two questions. $\endgroup$ Sep 4, 2019 at 20:21
  • $\begingroup$ @Michael I did (of course!). Could you be more specific about how they differ? $\endgroup$
    – whuber
    Sep 4, 2019 at 20:22
  • $\begingroup$ @whuber : The other one is asking how to define the likelihood function (not the density) for a mixture of continuous and discrete distributions. This one is not really about likelihood functions at all. The poster seems to think it is, but it's really about joint densities. The poster is familiar with the identity $\Pr(A\cap B) = \Pr(A)\Pr(B)$ when $A,B$ are independent events, and it's obvious how to apply that to a joint probability mass function, since the values of that function actually are probabilities. But he values of a density function of a continuous random$\,\ldots\qquad$ $\endgroup$ Sep 4, 2019 at 20:26
  • 1
    $\begingroup$ @Michael I see that the answers differ, but--despite the OP's responses to your answer--I remain unconvinced that these two threads are at all different. This question in its current form explicitly focuses on the likelihood and can readily be interpreted as wondering why it is valid to multiply probability densities to compute it. That's how I interpret the duplicate, too. I'm certainly open to changing my mind if the question were edited to rule out different interpretations. $\endgroup$
    – whuber
    Sep 4, 2019 at 20:38
  • 1
    $\begingroup$ @Michael, I don't see those issues as being irrelevant, but rather as generalizing the current question. It comes down to whether one perceives this question as being about multiplying densities or about treating densities as if they were probabilities in forming likelihoods. $\endgroup$
    – whuber
    Sep 4, 2019 at 21:07

1 Answer 1

5
$\begingroup$

If $f$ is a probability density of a random variable $X$, then we have $$ \Pr(X\in A) = \int\limits_A f(x)\, dx. $$ Now suppose $X$ and $Y$ are independent random variables with respective densities $f$ and $g.$ Then \begin{align} \Pr(X\in A\ \&\ Y\in B) & = \Pr(X\in A)\Pr(Y\in B) \\[10pt] & = \int\limits_A f(x)\,dx \cdot \int\limits_B g(x)\, dx \\[10pt] & = \int\limits_A f(x)\,dx \cdot \int\limits_B g(y)\, dy \end{align} because a bound variable can be freely re-named as long as the new name is not already taken. And you in an integral $\int_A \cdot\cdots \, dx$ with respect to $x,$, anything not depending on $x$ is a "constant", so you can write $\int_A f(x)\,dx \cdot 5 = \int_A f(x)\cdot5\, dx,$ etc., and accordingly we have \begin{align} & = \int\limits_A f(x) \left( \int\limits_B g(y)\, dy \right) \, dx \\[10pt] \text{and then } & = \int\limits_A \left( \int\limits_B f(x)g(y) \, dy \right) \, dx \tag 1 \\[10pt] & = \iint\limits_{A\times B} f(x) g(y) \, d(x,y) \text{ by Tonelli's theorem}. \tag 2 \end{align} Now, what does this mean? The "double integral" $$ \iint\limits_{A\times B} \cdots \, d(x,y) $$ is an integral with respect to the $2$-dimensional measure in the plane, in which the measure of a region is its area. The "iterated integral" $$ \int\limits_A \left( \int\limits_B \cdots\, dy \right) \, dx $$ involves two integrals with respect to $1$-dimensional measure in the line, in which the measure of an integral is its length. Tonelli's theorem says that as long as the function getting integrated is everywhere non-negative, then the values of the iterated integral in line $(1)$ and the double integral in line $(2)$ are the same (regardless of whether the value is a finite number or $+\infty$). (There is also the related Fubini's theorem, which says that if the double integral of the absolute value of the function is finite, then iterated integral and the double integral are equal.)

So now we have $$ \Pr(X\in A\ \&\ Y\in B) = \iint\limits_{A\times B} f(x) g(y) \, d(x,y). $$ Therefore the function $(x,y) \mapsto f(x) g(y)$ behaves like the density function of the random pair $(X,Y),$ at least as far as product sets like $A\times B$ are concerned. Does it still behave like the density of $(X,Y)$ if instead we integrate over, for example a disk in the plane? Here one would need to show that if it works for things like $A\times B,$ then it also works for things like a disk and for all other "measurable sets" in the plane. That's a fairly hairy argument, but if you've got that, then we're done.

$\endgroup$
9
  • $\begingroup$ No wonder the textbooks alway seem gloss over this point. $\endgroup$
    – rhody
    Sep 4, 2019 at 4:46
  • $\begingroup$ @rhody : There's also a heuristic argument, about like this: Thing of $dx$ and $y$ as infinitely small increments of $x$ and $y.$ Then $$ \Pr(x< X < x+dx\ \&\ y < Y < y+dy) = \big( f(x)\,dx\big)\big( g(y)\, dy\big) = f(x) g(y) \, d(x,y). $$ This sort of heuristic can be very useful even if it falls short of logical rigor. $\qquad$ $\endgroup$ Sep 4, 2019 at 4:50
  • $\begingroup$ @rhody : If you have any concerns about the comment about renaming bound variables, think about how it applies to this example: $$ \begin{align} & (1^2 + 2^2 + 3^2)(7^2 + 8^2 + 9^2) \\ {} \\ = {} & \left( \sum_{i=1}^3 i^2 \right)\left( \sum_{i=7}^9 i^2 \right) \\ {} \\ = {} & \left( \sum_{i=1}^3 i^2 \right)\left( \sum_{j=7}^9 j^2 \right) \quad \big( \text{re-naming a bound variable} \big) \\ {} \\ = {} & \sum_{i=1}^3 \left( i^2 \left( \sum_{j=7}^9 j^2 \right) \right) \\ {} \\ = {} & \sum_{i=1}^3 \sum_{j=7}^9 i^2 j^2 \end{align} $$ $\endgroup$ Sep 4, 2019 at 5:00
  • $\begingroup$ @rhody : $\quad \uparrow \quad$ I've made some corrections in the comment above since I first posted it, so look at it now. $\qquad$ $\endgroup$ Sep 4, 2019 at 5:07
  • $\begingroup$ I have been trying to think of it in terms of vertical slices, $f(x_i) \delta x_i$ of the pdf but hadn't got very far. I will need to study your arguments more closely. $\endgroup$
    – rhody
    Sep 4, 2019 at 5:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.