9
$\begingroup$

What does it mean to say that "Likelihood Function is Minimal Sufficient"? Is this a general statement, or does it apply to only exponential family of distributions?

I think I understand the concept of sufficient statistics, and minimal sufficient statistics.

Likelihood Function on the other hand is a function of the parameter with data plugged-in. We cannot calculate its value without knowing the parameter. It is different than say "sample mean" statistic for the mean parameter of a Normal distribution, which yields a concrete number. How is this a statistic then?

Then there is this lecture note, on section 2

http://www.stat.cmu.edu/~larry/=stat705/Lecture6.pdf

where Prof. Wasserman gets philosophical about the issue which confused me even more.

$\endgroup$
1

2 Answers 2

24
+100
$\begingroup$

Consider an observable data vector $\mathbf{x} = (x_1,...,x_n) \in \mathscr{X}$ with a joint-distribution that is indexed by the parameter $\theta \in \Theta$. It is possible to establish that "the likelihood function is minimal sufficient" in a sense described more specifically below.

Your misgivings here occur because you say that the likelihood is a function that depends on the parameter, and so we cannot calculate its value without the parameter. (This presumably means that it is not even a statistic, let alone a minimal sufficient statistic!) What you point out is true, but it misses the point, because the result asserted in your question is not that the value of the likelihood function at a particular point is minimal sufficient, but that the function is minimal sufficient. We will see below that when we talk about minimal sufficiency of "the likelihood function" we are really referring to the latter in its broadest sense, as a function of the data vector and the parameter.


Some preliminary notes on what a function is and is not: Before we get to the likelihood function, it is worth noting some general mathematical aspects of functions that are glossed over in early schooling, but which become important once you begin to study higher mathematics. In general, a function $f$ is a mapping from a domain $\mathscr{X}$ to a codomain $\mathscr{Y}$. To stress this meaning, we sometimes write the function as $f: \mathscr{X} \rightarrow \mathscr{Y}$. If we take any point $x \in \mathscr{X}$ then we can obtain the corresponding value of the function $f(x) \in \mathscr{Y}$ at this point. Nonetheless, there is a difference between the value $f(x)$ and the function $f$ --- they are not the same mathematical object.

In fact, it is possible to consider the function itself as a point in a larger function-space. Specifically, we can form the space $\mathscr{Y}^\mathscr{X}$ which contains all possible functions from the domain $\mathscr{X}$ to the codomain $\mathscr{Y}$, and we then have $f \in \mathscr{Y}^\mathscr{X}$ --- i.e., the function $f$ is a point in the function-space $\mathscr{Y}^\mathscr{X}$. Since the function can be considered as a point and the function value at a particular input can also be considered as a point, you can now see the distinction in these things more clearly --- for any $x \in \mathscr{X}$ the value $f(x)$ is a point in the space $\mathscr{Y}$ but the function $f$ is a point in the function-space $\mathscr{Y}^\mathscr{X}$ (which is typically a much larger space).

In high school mathematics, this distinction is sometimes glossed over with an abuse of notation by introducing functions with like this: "Let $f(x)$ be a function of $x$ with such-and-such properties...". In that context, the $x$ is treated as "generic" and so the object $f(x)$ is treated as a notation for the entire function, and not just its value at a particular point. Most mathematics textbooks used in high schools use this kind of notation/terminology, so there is no notational effort made to distinguish the function itself from its value at a particular point (though these are sometimes distinguished in the prose). At university level (undergraduate and postgraduate) most mathematics textbooks now begin to bring this distinction into the notation --- they start saying things like "Let $f$ be a function..." instead of "Let $f(x)$ be a function...". This is to get you used to the idea of distinguishing a function from its value at a particular input point, which becomes an important distinction in various higher mathematical results.


The mapping from the support of the data to the likelihood function is a sufficient statistic: For a particular observed instance of the data vector $\mathbf{x}$, the likelihood function is the mapping $L_\mathbf{x}: \Theta \rightarrow [0, \infty)$. It is a mapping that maps each value of $\theta$ to a real output, but the function itself is fixed by the specified value of $\mathbf{x}$ and the domain $\Theta$. (This is just an instance where we have to be careful to distinguish a function from its value at a specific argument value.) Now, let $\mathscr{T} \equiv [0, \infty)^\Theta$ be the space of all mappings from the parameter space $\Theta$ to the non-negative real numbers. For each data vector $\mathbf{x} \in \mathscr{X}$ there is a corresponding likelihood function $L_\mathbf{x} \in \mathscr{T}$, and so we may consider that there is a mapping $T: \mathscr{X} \rightarrow \mathscr{T}$ that maps each possible observable data vector onto its corresponding likelihood function (i.e., we have $T(\mathbf{x}) = L_\mathbf{x}$). This mapping is "the likelihood function" in its broadest sense, where we have not yet specified the observed data vector.

You can see that this function $T$ is a mapping from the domain of possible data vectors to a fixed codomain, so it is a statistic. This is the sense in which we can consider the "likelihood function" to be a statistic. That is, the "likelihood function" can be considered as a statistic, if we have not yet specified the observed data vector (so that it is a function of this data vector), and we are looking at the object as a function of the parameter, rather than looking at an outcome value that accrues from a particular parameter value. With this elaboration, we now understand that the function $T$ is "the likelihood function" in this broadest sense.

Theorem 1: The function $T$ is sufficient for $\theta$.

Proof: For all $\theta \in \Theta$ we define the function $g_\theta : \mathscr{T} \rightarrow \mathbb{R}$ by $g_\theta(f) = f(\theta)$. Using the fact that the likelihood function is proportional to the sampling density, we can write the sampling density as: $$\begin{equation} \begin{aligned} f_\theta(\mathbf{x}) &= h(\mathbf{x}) L_\mathbf{x}(\theta) \\[6pt] &= h(\mathbf{x}) g_\theta(L_\mathbf{x}) \\[6pt] &= h(\mathbf{x}) g_\theta(T(\mathbf{x})). \\[6pt] \end{aligned} \end{equation}$$ This establishes the Fisher-Neyman factorisation, which establishes the theorem. $\blacksquare$


Demonstrating minimal sufficiency: The above mapping $T$ is a sufficient statistic, but it is not generally minimal sufficient for $\theta$. To ensure minimal sufficiency, we need to narrow things down further than this, since the likelihood function is defined non-arbitrarily only up to proportionality. For this reason, if we want to get minimal sufficiency, we will need to consider two proportionate likelihood functions $L_\mathbf{x}$ and $L_{\mathbf{x}'}$ to be "the same" function. In the linked lecture notes, you will see that this is done by defining an equivalence relation $\sim$ as follows:

$$\mathbf{x} \sim \mathbf{x}' \quad \quad \quad \quad\iff \quad \quad \quad \quad L_\mathbf{x} \propto L_{\mathbf{x}'}.$$

This equivalence relation says that two likelihood functions $L_\mathbf{x}$ and $L_{\mathbf{x}'}$ are "the same" when proportionate. It induces a corresponding partition on $\mathscr{X}$ that separates the observable data vectors into sets that yield "the same" likelihood function (in the sense of proportionality).

Theorem 2: The partition $\sim$ is the minimal sufficient partition for $\theta$.

This theorem is left as an exercise in the linked lecture notes. Proof can be done by first establishing the sufficiency of the partition (which is assured by Theorem 1 above), and then showing that if you coarsen the partition, sufficiency is lost. Since this is set as an exercise in the notes for the current question, I will not prove the result here. Hopefully the above information gets you started, and more clearly establishes the statistic that you are dealing with.

The fact that the likelihood function (conceived as a statistic) is minimal sufficient, really should not be surprising. Indeed, it is practically a tautology, since sufficiency can be framed as a condition in terms of the likelihood function. Sufficiency means that the statistic captures all the required information about the indexing parameter. Since the likelihood is proportional to the sampling density, it is hardly surprising that it does this. Minimal sufficiency means that it does so without additional information, which is easily achieved by partitioning down to a level where the statistic/partition of interest is set by proportionality to the likelihood function.

The reason for the importance of minimal sufficiency is that every minimal sufficient statistic induces the same partition induced by the equivalence relation on the likelihood function. This means that every minimal sufficient statistic is a stand-in for the equivalence relation induced by proportionality of the likelihood function.

$\endgroup$
2
  • 1
    $\begingroup$ Nice answer. Is there a particular importance of the fact that likelihood function is minimal sufficient? $\endgroup$ Commented Apr 30, 2019 at 11:04
  • 5
    $\begingroup$ Since the minimal sufficient partition is unique, it means that every minimal sufficient statistic induces the same partition induced by the equivalence relation on the likelihood function. In short, every minimal sufficient statistic is a stand-in for the equivalence relation induced by proportionality of the likelihood function. $\endgroup$
    – Ben
    Commented Apr 30, 2019 at 11:11
5
$\begingroup$

There's an unobservable parameter, or a tuple of such parameters. Call it $\theta.$ The value of the likelihood function $L$ at $\theta$ is $L(\theta)$ and is similarly unobservable. It is not a statistic, let alone a sufficient statistic.

When it is said that the likelihood function $L$ is a minimal sufficient statistic, what is meant is not that the unobservable value $L(\theta)$ of the likelihood function at some unobservable value $\theta$ of the parameter is a minimal sufficient statistic, but rather that the function $L$ as a whole is a minimial sufficient statistic. The function $L$ is observable, in that you know which function it is, after computing it based on the data.

That it is a sufficient statistic means that for any two values $U,$ $V$ of the data, if the likelihood function given observations $U$ is the same as that given $V,$ then the conditional distribution of the data given which function $L$ is, is the same if the observed data are $V$ as if they are $U.$

$\endgroup$
11
  • 1
    $\begingroup$ With my limited knowledge of mathematics, I would say a function is some sort of infinite dimensional vector. From this perspective how is it a minimal sufficient statistic, when obviously we have in many cases finite dimensional sufficient statistic at hand? $\endgroup$ Commented Nov 2, 2018 at 6:49
  • 1
    $\begingroup$ @CowboyTrader : To say that a sufficient statistic $S$ is minimal means that for every sufficient statistic $T,$ the value of $S$ is determined by the value of $T$. It's not a matter of dimension. $\endgroup$ Commented Nov 2, 2018 at 23:08
  • 2
    $\begingroup$ @CowboyTrader : You're not using the word "sample" correctly. You should write of the size of the sample, which is the number of observations in the sample, rather than of the "number of samples". $\endgroup$ Commented Nov 3, 2018 at 19:07
  • 1
    $\begingroup$ Don Fraser (Toronto) has been arguing since 1960 that indeed the likelihood function is sufficient but I have the same issue with a function being an infinite dimensional object. $\endgroup$
    – Xi'an
    Commented Nov 5, 2018 at 17:32
  • 3
    $\begingroup$ @Xi'an : For a family of distributions parametrized by finitely many scalar parameters, there is only a finite-dimensional space of possible likelihood functions. $\endgroup$ Commented Nov 6, 2018 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.