4
$\begingroup$

In statistics, the likelihood function (often simply called the likelihood) is formed from the joint probability of a sample of data given a set of model parameter values; it is viewed and used as a function of the parameters given the data sample.[a]

I heard that it's necessary to define sigma algebras to define probability spaces.

Are likelihood function also defined requiring the idea of sigma algebras? Is there a probability space analog to likelihood spaces if those exist?

Why is there only a subset of mathematics called probability theory and not likelihood theory also? Why don't mathematicians study the measure theoretic properties of likelihood functions and likelihood spaces if those exist and call it likelihood theory?

The likelihood function seems to have the same properties as the probability function with some important exceptions so I was wondering whether the sigma-algebra properties would be inherited.

$\endgroup$
  • 3
    $\begingroup$ From a measure-theoretic standpoint, a likelihood is just a Radon-Nikodym derivative of a probability distribution with respect to some dominating measure (e.g., Lebesgue measure for absolutely continuous distributions or counting measure for discrete distributions) $\endgroup$ – Artem Mavrin Jan 18 at 22:30
5
$\begingroup$

Great question. The answer is that a likelihood function is just a function of random variables, so that the probability space that encapsulates all the probabilistic information of interest about those random variables (including the $\sigma$-fields you mention) is often still all that is required for a function of those random variables.

For example let $X_{1},X_{2}$ be random variables on the same underlying probability space $(\Omega,\mathscr{F})$ where $\mathscr{F}$ is a $\sigma$-field of subsets of $\Omega$. Then by definition $X_{i}:\Omega\longrightarrow\mathbb{R}$ for $i=1,2$ is a function that satisfies $X_{i}^{-1}(B)=\{\omega\in\Omega:X_{i}(\omega)\in B\}\in\mathscr{F}$ for all $B\in\mathscr{R}$ where $\mathscr{R}$ is the $\sigma$-field of subsets of $\mathbb{R}$. This is referred to as $X_{i}$ being $\mathscr{F}/\mathscr{R}$ measurable, "measurable" since $P$ (the probability measure) assigns a probability to the "event" $X_{i}\in B$ as $Pr[X_{i}\in B]=P[X_{i}^{-1}(B)]$. Thus a random variable is a "measurable mapping" between the measure spaces $(\Omega,\mathscr{F})$ and $(\mathbb{R},\mathscr{R})$.

Consider the (fixed) function $f:\mathbb{R}\times\mathbb{R}\longrightarrow\mathbb{R}$, $f(y,z)\mapsto y+z+37$ and use $X_{1}$ and $X_{2}$ as random arguments: $f(X_{1}(\omega_{1}),X_{2}(\omega_{2}))\mapsto X_{1}(\omega_{1})+X_{1}(\omega_{2})+37$. The question you ask of likelihood functions can be asked of any function. Can we assign probabilities to the event $[f(X_{1}(\omega_{1}),X_{2}(\omega_{2}))]^{-1}(B)=\{(\omega_{1},\omega_{2})\in \Omega\times\Omega:(X_{1}(\omega_{1})+X_{1}(\omega_{2})+37)\in B\}$ for all $B\in\mathscr{R}$ using the original space $(\Omega,\mathscr{F})$? The answer is yes if $\{(\omega_{1},\omega_{2})\in \Omega\times\Omega:(X_{1}(\omega_{1})+X_{2}(\omega_{2})+37)\in B\}\in\mathscr{F}$ since $P$ can assign probabilities to any $F\in\mathscr{F}$. If $f$ is continuous (here it is) then $[f(X_{1}(\omega_{1}),X_{2}(\omega_{2}))]^{-1}\in\mathscr{F}$ indeed holds. Thus $Pr[f(X_{1},X_{2})\in B]=P\left\{[f(X_{1},X_{2})]^{-1}(B)\right\}$

Thus viewing $f$ not as the mapping $f:\mathbb{R}\times \mathbb{R}\longrightarrow\mathbb{R}$, $f(y,z)\mapsto y+z+37$ but instead as $f:\Omega\times \Omega\longrightarrow\mathbb{R}$, $f(X_{1}(\omega_{1}),X_{2}(\omega_{2}))\mapsto X_{1}(\omega_{1})+X_{2}(\omega_{2})+37$ we see that $f$ is $\mathscr{F}/\mathscr{R}$ measurable - i.e. it is a random variable. Furthermore the probability space is again $(\Omega,\mathscr{F})$, so nothing new (probabilistically) is needed. Other properties aside from continuity also give measurable mappings, but the main point is that it will not do to take any function.

Another way of viewing all this is as the composition of two functions $f\circ W$, for $W=(X_{1},X_{2}):\Omega\times\Omega\longrightarrow\mathbb{R}\times\mathbb{R}$. We have $W$ being $\mathscr{F}/(\mathscr{R}\times \mathscr{R})$ measurable and $f$ being $(\mathscr{R}\times \mathscr{R})\times\mathscr{R}$ measurable. There is a result that states in these circumstnces that $f\circ W$ is then $\mathscr{F}/\mathscr{R}$ measurable

To your likelihood question now. Let the joint density for $W=(X_{1},X_{2})$ be denoted $f(y,z|\theta)$ for parameter $\theta\in\Theta$. By definition $W$ is $\mathscr{F}/(\mathscr{R}\times \mathscr{R})$ measurable (since $W$ is a random variable) and $f:\mathbb{R}\times \mathbb{R}\longrightarrow \mathbb{R}$ is $(\mathscr{R}\times \mathscr{R})\times\mathscr{R}$ measurable (since $f$ is the density for the distribution of $W$) so as above $f\circ W\mapsto f(X_{1}(\omega_{1}),X_{2}(\omega_{2})|\theta)$ is then $\mathscr{F}/\mathscr{R}$ measurable and $Pr[f(X_{1},X_{2}|\theta)\in B]=P\left\{[f(X_{1},X_{2}|\theta)]^{-1}(B)\right\}$ for all $B\in\mathscr{R}$. So again the original probability measure $P$ and $\sigma$-field $\mathscr{F}$ are enough to describe the randomness in $f$.

Edit: posted an incomplete answer since OP just removed this from maths stackexchange – I will now try and finish before this is removed again!

Continuing

It is worth mentioning the relationship between the distribution (law) for $W$ and $f$ the density. Let $\mu_{\theta}$ be the probability law for $(X_{1},X_{2})$ for a $\theta\in\Theta$ then by definition $Pr[(X_{1},X_{2})\in R|\theta]=\mu_{\theta}(R)=\int_{R}f(y,z|\theta)d(y,z)$ for all $R\in\mathscr{R}\times\mathscr{R}$ (this is the $\sigma$-field of subsets of $\mathbb{R}\times\mathbb{R}$) which shows that probabilities of $f$ being in the linear Borel sets $\mathscr{R}$, i.e $Pr[f(X_{1},X_{2}|\theta)\in B]$ are not the same concept as the probabilities of $W$ being in the $2$-dimensional Borel sets $\mathscr{R}\times\mathscr{R}$, i.e $Pr[(X_{1},X_{2})\in R|\theta]$. Again $f(X_{1},X_{2}|\theta)$ is just a random function to which $P$ can assign probabilities to (of $f$ being in certain sets).

The likelihood function $L(\theta|x_{1},x_{2})$ is, as you say, is just $f(x_{1},x_{2}|\theta)$ viewed as a function of $\theta\in\Theta$ rather than $W\in\mathscr{R}\times \mathscr{R}$. Nonetheless this "viewpoint" is for one realisation of $W=w$. When inference is performed we think of repeated realisations of $W$ (in a frequentist inference setting anyway), and so we write $L(\theta|X_{1},X_{2})=f(X_{1},X_{2}|\theta)$. For fixed $\theta$ this is exactly as stated above - a $\mathscr{F}/\mathscr{R}$ measurable function. Thus for $\theta',\theta''\in\Theta$, $L(\theta'|X_{1},X_{2})$ and $L(\theta''|X_{1},X_{2})$ are both $\mathscr{F}/\mathscr{R}$ measurable functions that can be described, probabilistically, with $P$ and $\mathscr{F}$, so no new probability space needed.

As answered above by another user, the key difference to $f$ is that the analagous result to

$Pr[(X_{1},X_{2})\in \mathbb{R}^{2}|\theta]=\mu_{\theta}(\mathbb{R}^{2})=\int_{\mathbb{R}^{2}}f(y,z|\theta)d(y,z)=1$

does not hold for $L$, that is for any observed $x_{1},x_{2}$

$\int_{\Theta}L(\theta|x_{1},x_{2})d(\theta)\not=1$

$\endgroup$
3
$\begingroup$

In statistics, a likelihood function is a function of the unknown parameter; it is not a function of the random variable. The random variables are assumed to have already been observed, and the likelihood function only sees them as constants.

So, while certainly the statistical model for how these data were observed must follow all of the mathematical axioms of a probability space, the value of the likelihood function need not be a probability. Certainly, for a continuous random variable this is the case as the probability of the observed data taking exactly the value it did is zero regardless of the parameter value. In fact, the term "likelihood" is used specifically because the value of the joint density for a given parameter value need not be a probability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.