# How to rigorously define the likelihood?

The likelihood could be defined by several ways, for instance :

• the function $L$ from $\Theta\times{\cal X}$ which maps $(\theta,x)$ to $L(\theta \mid x)$

• the random function $L(\cdot \mid X)$

• we could also consider that the likelihood is only the "observed" likelihood $L(\cdot \mid x^{\text{obs}})$

• in practice the likelihood brings information on $\theta$ only up to a multiplicative constant, hence we could consider the likelihood as an equivalence class of functions rather than a function

Another question occurs when considering change of parametrization: if $\phi=\theta^2$ is the new parameterization we commonly denote by $L(\phi \mid x)$ the likelihood on $\phi$ and this is not the evaluation of the previous function $L(\cdot \mid x)$ at $\theta^2$ but at $\sqrt{\theta}$. This is an abusive but useful notation which could cause difficulties to beginners if it is not emphasized.

What is your favorite rigorous definition of the likelihood ?

In addition how do you call $L(\theta \mid x)$ ? I usually say something like "the likelihood on $\theta$ when $x$ is observed".

EDIT: In view of some comments below, I realize I should have precised the context. I consider a statistical model given by a parametric family $\{f(\cdot \mid \theta), \theta \in \Theta\}$ of densities with respect to some dominating measure, with each $f(\cdot \mid \theta)$ defined on the observations space ${\cal X}$. Hence we define $L(\theta \mid x)=f(x \mid \theta)$ and the question is "what is $L$ ?" (the question is not about a general definition of the likelihood)

-
(1) Because $\int L(\theta|x)dx = 1$ for all $\theta$, I believe even the constant in $L$ is defined. (2) If you think of parameters like $\phi$ and $\theta$ as merely being coordinates for a manifold of distributions, then change of parameterization has no intrinsic mathematical meaning; it's merely a change of description. (3) Native English speakers would more naturally say "likelihood of $\theta$" rather than "on." (4) The clause "when $x$ is observed" has philosophical difficulties, because most $x$ will never be observed. Why not just say "likelihood of $\theta$ given $x$"? –  whuber Jun 2 '12 at 16:19
@whuber: For (1), I don't think the constant is well-defined. See ET Jaynes's book where he writes: "that a likelihood is not a probability because its normalization is arbitrary." –  Neil G Jun 2 '12 at 17:31
You appear to be confusing two kinds of normalization, Neil: Jaynes was referring to normalization by integration over $\theta$, not $x$. –  whuber Jun 2 '12 at 17:32
@whuber: I don't think a scaling factor will matter for the Cramer-Rao bound because changing $k$ adds a constant amount to the log-likelihood, which then disappears when the partial derivative is taken. –  Neil G Jun 3 '12 at 0:04
I agree with Neil, I do not see any application where the constant plays a role –  Stéphane Laurent Jun 3 '12 at 6:14

Your third item is the one I have seen the most often used as rigorous definition.

The others are interesting too (+1). In particular the first is appealing, with the difficulty that the sample size not being (yet) defined, it is harder to define the "from" set.

To me, the fundamental intuition of the likelihood is that it is a function of the model + its parameters, not a function of the random variables (also an important point for teaching purposes). So I would stick to the third definition.

The source of the abuse of notation is that the "from" set of the likelihood is implicit, which is usually not the case for well defined functions. Here, the most rigorous approach is to realize that after the transformation, the likelihood relates to another model. It is equivalent to the first, but still another model. So the likelihood notation should show which model it refers to (by subscript or other). I never do it of course, but for teaching, I might.

Finally, to be consistent with my previous answers, I say the "likelihood of $\theta$" in your last formula.

-
Thanks. And what is your advice about the equality up to a multiplicative constant ? –  Stéphane Laurent Jun 2 '12 at 12:15
Personally I prefer to call it up when needed rather than hard code it in the definition. And think that for model selection/comparison this 'up-to-a-multiplicative-constant' equality does not hold. –  gui11aume Jun 2 '12 at 12:21
Ok. Concerning the name, you could imagine you discuss about the likelihoods $L(\theta\mid x_1)$ and $L(\theta\mid x_2)$ for two possibles observations. In such a case, would you say "the likelihood of $\theta$ when $x_1$ observed", or "the likelihood of $\theta$ for the observation $x_1$", or something else ? –  Stéphane Laurent Jun 2 '12 at 14:11
If you re-parametrize your model with $\phi = \theta^2$ you actually compute the likelihood as a composition of functions $L(.|x) \circ g(.)$ where $g(y) = y^2$. In this case, $g$ goes from $R$ to $R^+$ so the set of definition (mentioned as "from" set) of the likelihood is no longer the same. You could call the first function $L_1(.|)$ and the second $L_2(.|)$ because they are not the same functions. –  gui11aume Jun 2 '12 at 14:57
How is the third definition rigorous? And what is the problem with the sample size not being defined? Since we say $P(x_1, x_2, \dotsc, x_n \mid \theta)$, which naturally brings into existence a corresponding sigma algebra for the sample space $\Omega^n$, why can't we have the parallel definition for likelihoods? –  Neil G Jun 2 '12 at 16:13

I think I would call it something different. Likelihood is the probability density for the observed x given the value of the parameter $θ$ expressed as a function of $θ$ for the given $x$. I don't share the view about the proportionality constant. I think that only comes into play because maximizing any monotonic function of the likelihood gives the same solution for $θ$. So you can maximize $cL(θ∣x)$ for $c>0$ or other monotonic functions such as $\log(L(θ∣x))$ which is commonly done.

-
Not only the maximization: the up-to-proportionality also comes into play in the likelihood ratio notion, and in Bayes formula for Bayesian statistics –  Stéphane Laurent Jun 2 '12 at 13:41
Yes Stephane. Thank you. –  Michael Chernick Jun 2 '12 at 14:36
I thought someone might downvote my answer. But I think it is quite reasonable to define likelihood this way as a definitive probability without calling anything proprotional to it a likelihood. @StéphaneLaurent to your comment about priors, if the function is integrable it can be normalized to a density. The posterior is proportional to the likelihood times the prior. Since the posterior must be normalized by dividing by an integral we might as well specify the prior to be the distribution. It is only in an extended sense that this gets applied to improper priors. –  Michael Chernick Jun 2 '12 at 16:35
I'm not quite sure why someone would downvote this answer. It seems you are trying to respond more to the OP's second and questions than the first. Perhaps that was not entirely clear to other readers. Cheers. :) –  cardinal Jun 2 '12 at 16:50
@Stephane: If you'd like, please consider posting your other question directly in French. We have several native French speakers on this site that likely would help translate any passages you're unsure about. This includes a moderator and also an editor of one of the very top English-language statistics journals. I look forward to the question. –  cardinal Jun 2 '12 at 17:35

Here's an attempt at a rigorous mathematical definition:

Let $X: \Omega \to \mathbb R^n$ be a random vector which admits a density $f(x | \theta_0)$ with respect to some measure $\nu$ on $\mathbb R^n$, where for $\theta \in \Theta$, $\{f(x|\theta): \theta \in \Theta\}$ is a family of densities on $\mathbb R^n$ with respect to $\nu$. Then, for any $x \in \mathbb R^n$ we define the likelihood function $L(\theta | x)$ to be $f(x | \theta)$; for clarity, for each $x$ we have $L_x : \Theta \to \mathbb R$. One can think of $x$ to be a particular potential $x_{obs}$ and $\theta_0$ to be the "true" value of $\theta$.

1. The definition is robust enough to handle discrete, continuous, and other sorts of families of distributions for $X$.
2. We are defining the likelihood at the level of density functions instead of at the level of probability distributions/measures. The reason for this is that densities are not unique, and it turns out that this isn't a situation where one can pass to equivalence classes of densities and still be safe: different choices of densities lead to different MLE's in the continuous case. However, in most cases there is a natural choice of family of densities that are desirable theoretically.
3. I like this definition because it incorporates the random variables we are working with into it and, by design since we have to assign them a distribution, we have also rigorously built in the notion of the "true but unknown" value of $\theta$, here denoted $\theta_0$. For me, as a student, the challenge of being rigorous about likelihood was always how to reconcile the real world concepts of a "true" $\theta$ and "observed" $x_{obs}$ with the mathematics; this was often not helped by instructors claiming that these concepts weren't formal but then turning around and using them formally when proving things! So we deal with them formally in this definition.
4. EDIT: Of course, we are free to consider the usual random elements $L(\theta | X)$, $S(\theta | X)$ and $\mathcal I(\theta | X)$ and under this definition with no real problems with rigor as long as you are careful (or even if you aren't if that level of rigor is not important to you).
-
Yes, I should have precised that I only consider statistical models given by a family $\{f(x|\theta): \theta \in \Theta\}$ of Radon-Nikodym derivatives ("densities") with respect to some dominating measure $\nu$, and my question is about the nature of $L$ defined by $L(\theta \mid x) = f(x \mid \theta)$. –  Stéphane Laurent Jun 2 '12 at 16:33
I do not get this measure theoretic remark about different versions of densities: the likelihood is a function of $\theta$ for the observed data $x_{obs}$. You cannot start changing the versions at $x_{obs}$ once $x_{obs}$ is observed... In other words, the probability that $x_{obs}$ belongs to a set where the versions differ is zero, because this set has measure zero... –  Xi'an Jun 16 '12 at 19:47
@Xi'an I think your definition is too limited to be generally useful. One uses likelihoods not only for estimation but also for theoretical considerations in which all possible outcomes have to be contemplated. That is why guy takes some care to assure there is some commonality among the densities within a family. –  whuber Jun 16 '12 at 19:56
@whuber: I would be most interested in an example where the version of the density function matters for estimation purposes... Even in a frequentist perspective. –  Xi'an Jun 16 '12 at 20:09
@Xi'an It begins to sound like you, guy, myself, and perhaps others may be operating with sufficiently different assumptions about the statistical setting, the objectives, and even the definitions of mathematical objects like densities that it may take quite a lot of discussion to straighten things out, and then I (strongly) suspect we will find no fundamental disagreement. This is a conversation that belongs in chat, because the commenting mechanism is too confining, laborious, (and distracting) to be helpful. –  whuber Jun 16 '12 at 20:17