7
$\begingroup$

What does it mean to say that "Likelihood Function is Minimal Sufficient"? Is this a general statement, or does it apply to only exponential family of distributions?

I think I understand the concept of sufficient statistics, and minimal sufficient statistics.

Likelihood Function on the other hand is a function of the parameter with data plugged-in. We cannot calculate its value without knowing the parameter. It is different than say "sample mean" statistic for the mean parameter of a Normal distribution, which yields a concrete number. How is this a statistic then?

Then there is this lecture note, on section 2

http://www.stat.cmu.edu/~larry/=stat705/Lecture6.pdf

where Prof. Wasserman gets philosophical about the issue which confused me even more.

$\endgroup$
10
+100
$\begingroup$

Consider an observable data vector $\mathbf{x} = (x_1,...,x_n) \in \mathscr{X}$ with a joint-distribution that is indexed by the parameter $\theta \in \Theta$. It is possible to establish that "the likelihood function is minimal sufficient", in a sense described more specifically below.

Your misgivings here occur because you say that the likelihood is a function that depends on the parameter, and so we cannot calculate its value without the parameter. (This presumably means that it is not even a statistic, let alone a minimal sufficient statistic!) What you point out is true, but it misses the point, because the result asserted in your question is not that the value of the likelihood function at a particular point is minimal sufficient, but that the function is minimal sufficient. We will see below that when we talk about minimal sufficiency of "the likelihood function" we are really referring to the latter in its broadest sense, as a function of the data vector.


The mapping from the support of the data to the likelihood function is a sufficient statistic: For a particular observed instance of the data vector $\mathbf{x}$, the likelihood function is the mapping $L_\mathbf{x}: \Theta \rightarrow [0, \infty)$. It is a mapping that maps each value of $\theta$ to a real output, but the function itself is fixed by the specified value of $\mathbf{x}$ and the domain $\Theta$. (This is just an instance where we have to be careful to distinguish a function from its value at a specific argument value.) Now, let $\mathscr{T} \equiv [0, \infty)^\Theta$ be the space of all mappings from the parameter space $\Theta$ to the non-negative real numbers. For each data vector $\mathbf{x} \in \mathscr{X}$ there is a corresponding likelihood function $L_\mathbf{x} \in \mathscr{T}$, and so we may consider that there is a mapping $T: \mathscr{X} \rightarrow \mathscr{T}$ that maps each possible observable data vector onto its corresponding likelihood function (i.e., we have $T(\mathbf{x}) = L_\mathbf{x}$). This mapping is "the likelihood function" in its broadest sense, where we have not yet specified the observed data vector.

You can see that this function $T$ is a mapping from the domain of possible data vectors to a fixed codomain, so it is a statistic. This is the sense in which we can consider the "likelihood function" to be a statistic. That is, the "likelihood function" can be considered as a statistic, if we have not yet specified the observed data vector (so that it is a function of this data vector), and we are looking at the object as a function of the parameter, rather than looking at an outcome value that accrues from a particular parameter value. With this elaboration, we now understand that the function $T$ is "the likelihood function" in this broadest sense.

Theorem 1: The function $T$ is sufficient for $\theta$.

Proof: For all $\theta \in \Theta$ we define the function $g_\theta : \mathscr{T} \rightarrow \mathbb{R}$ by $g_\theta(f) = f(\theta)$. Using the fact that the likelihood function is proportional to the sampling density, we can write the sampling density as: $$\begin{equation} \begin{aligned} f_\theta(\mathbf{x}) &= h(\mathbf{x}) L_\mathbf{x}(\theta) \\[6pt] &= h(\mathbf{x}) g_\theta(L_\mathbf{x}) \\[6pt] &= h(\mathbf{x}) g_\theta(T(\mathbf{x})). \\[6pt] \end{aligned} \end{equation}$$ This establishes the Fisher-Neyman factorisation, which establishes the theorem. $\blacksquare$


Demonstrating minimal sufficiency: The above mapping $T$ is a sufficient statistic, but it is not generally minimal sufficient for $\theta$. To ensure minimal sufficiency, we need to narrow things down further than this, since the likelihood function is defined non-arbitrarily only up to proportionality. For this reason, if we want to get minimal sufficiency, we will need to consider two proportionate likelihood functions $L_\mathbf{x}$ and $L_{\mathbf{x}'}$ to be "the same" function. In the linked lecture notes, you will see that this is done by defining an equivalence relation $\sim$ as follows:

$$\mathbf{x} \sim \mathbf{x}' \quad \quad \quad \quad\iff \quad \quad \quad \quad L_\mathbf{x} \propto L_{\mathbf{x}'}.$$

This equivalence relation says that two likelihood functions $L_\mathbf{x}$ and $L_{\mathbf{x}'}$ are "the same" when proportionate. It induces a corresponding partition on $\mathscr{X}$ that separates the observable data vectors into sets that yield "the same" likelihood function (in the sense of proportionality).

Theorem 2: The partition $\sim$ is the minimal sufficient partition for $\theta$.

This theorem is left as an exercise in the linked lecture notes. Proof can be done by first establishing the sufficiency of the partition (which is assured by Theorem 1 above), and then showing that if you coarsen the partition, sufficiency is lost. Since this is set as an exercise in the notes for the current question, I will not prove the result here. Hopefully the above information gets you started, and more clearly establishes the statistic that you are dealing with.

The fact that the likelihood function (conceived as a statistic) is minimal sufficient, really should not be surprising. Indeed, it is practically a tautology, since sufficiency can be framed as a condition in terms of the likelihood function. Sufficiency means that the statistic captures all the required information about the indexing parameter. Since the likelihood is proportional to the sampling density, it is hardly surprising that it does this. Minimal sufficiency means that it does so without additional information, which is easily achieved by partitioning down to a level where the statistic/partition of interest is set by proportionality to the likelihood function.

The reason for the importance of minimal sufficiency is that every minimal sufficient statistic induces the same partition induced by the equivalence relation on the likelihood function. This means that every minimal sufficient statistic is a stand-in for the equivalence relation induced by proportionality of the likelihood function.

$\endgroup$
2
  • $\begingroup$ Nice answer. Is there a particular importance of the fact that likelihood function is minimal sufficient? $\endgroup$ – Cagdas Ozgenc Apr 30 '19 at 11:04
  • 1
    $\begingroup$ Since the minimal sufficient partition is unique, it means that every minimal sufficient statistic induces the same partition induced by the equivalence relation on the likelihood function. In short, every minimal sufficient statistic is a stand-in for the equivalence relation induced by proportionality of the likelihood function. $\endgroup$ – Ben Apr 30 '19 at 11:11
5
$\begingroup$

There's an unobservable parameter, or a tuple of such parameters. Call it $\theta.$ The value of the likelihood function $L$ at $\theta$ is $L(\theta)$ and is similarly unobservable. It is not a statistic, let alone a sufficient statistic.

When it is said that the likelihood function $L$ is a minimal sufficient statistic, what is meant is not that the unobservable value $L(\theta)$ of the likelihood function at some unobservable value $\theta$ of the parameter is a minimal sufficient statistic, but rather that the function $L$ as a whole is a minimial sufficient statistic. The function $L$ is observable, in that you know which function it is, after computing it based on the data.

That it is a sufficient statistic means that for any two values $U,$ $V$ of the data, if the likelihood function given observations $U$ is the same as that given $V,$ then the conditional distribution of the data given which function $L$ is, is the same if the observed data are $V$ as if they are $U.$

$\endgroup$
11
  • 1
    $\begingroup$ With my limited knowledge of mathematics, I would say a function is some sort of infinite dimensional vector. From this perspective how is it a minimal sufficient statistic, when obviously we have in many cases finite dimensional sufficient statistic at hand? $\endgroup$ – Cagdas Ozgenc Nov 2 '18 at 6:49
  • 1
    $\begingroup$ @CowboyTrader : To say that a sufficient statistic $S$ is minimal means that for every sufficient statistic $T,$ the value of $S$ is determined by the value of $T$. It's not a matter of dimension. $\endgroup$ – Michael Hardy Nov 2 '18 at 23:08
  • 2
    $\begingroup$ @CowboyTrader : You're not using the word "sample" correctly. You should write of the size of the sample, which is the number of observations in the sample, rather than of the "number of samples". $\endgroup$ – Michael Hardy Nov 3 '18 at 19:07
  • 1
    $\begingroup$ Don Fraser (Toronto) has been arguing since 1960 that indeed the likelihood function is sufficient but I have the same issue with a function being an infinite dimensional object. $\endgroup$ – Xi'an Nov 5 '18 at 17:32
  • 2
    $\begingroup$ @Xi'an : For a family of distributions parametrized by finitely many scalar parameters, there is only a finite-dimensional space of possible likelihood functions. $\endgroup$ – Michael Hardy Nov 6 '18 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.