# Intuitive understanding of the Halmos-Savage theorem

The Halmos-Savage theorem says that for a dominated statistical model $$(\Omega, \mathscr A, \mathscr P)$$ a statistic $$T: (\Omega, \mathscr A, \mathscr P)\to(\Omega', \mathscr A')$$ is sufficient if (and only if) for all $$\{P \in \mathscr{P} \}$$ there is a $$T$$-measurable version of the Radon Nikodym derivative $$\frac{dP}{dP*}$$ where $$dP*$$ is a privileged measure such that $$P*=\sum_{i=1}^\infty P_i c_i$$ for $$c_i >0, \sum _{i=1}^\infty c_i =1$$ and $$P_i \in \mathscr P$$.

I have tried to get an intuitive grasp why the theorem is true but I did not succeed, so my question is whether there is an intuitive way to understand the theorem.

• I believe I have the correct link here. Please check & remove it if I made a mistake. – gung - Reinstate Monica Oct 10 '18 at 19:16
• Maybe help the reader with terminology, e.g., define "dominated statistical models", "$T$-measurability" and "privileged measures? – Carl Oct 20 '18 at 19:39

# A Technical Lemma

I'm not sure how intuitive this is, but the main technical result underlying your statement of the Halmos-Savage Theorem is the following:

Lemma. Let $$\mu$$ be a $$\sigma$$-finite measure on $$(S, \mathcal{A})$$. Suppose that $$\aleph$$ is a collection of measures on $$(S, \mathcal{A})$$ such that for every $$\nu \in \aleph$$, $$\nu \ll \mu$$. Then there exists a sequence of nonnegative numbers $$\{c_i\}_{i=1}^\infty$$ and a sequence of elements of $$\aleph$$, $$\{\nu_i\}_{i=1}^\infty$$ such that $$\sum_{i=1}^\infty c_i = 1$$ and $$\nu \ll \sum_{i=1}^\infty c_i \nu_i$$ for every $$\nu \in \aleph$$.

This is taken verbatim from Theorem A.78 in Schervish's Theory of Statistics (1995). Therein he attributes it to Lehmann's Testing Statistical Hypotheses (1986) (link to the third edition), where the result is attributed to Halmos and Savage themselves (see Lemma 7). Another good reference is Shao's Mathematical Statistics (second edition, 2003), where the relevant results are Lemma 2.1 and Theorem 2.2.

The lemma above states that if you start with a family of measures dominated by a $$\sigma$$-finite measure, then in fact you can replace the dominating measure by a countable convex combination of measures from within the family. Schervish writes before stating Theorem A.78,

"In statistical applications, we will often have a class of measures, each of which is absolutely continuous with respect to a single $$\sigma$$-finite measure. It would be nice if the single dominating measure were in the original class or could be constructed from the class. The following theorem addresses this problem."

# A Concrete Example

Suppose we take a measurement of a quantity $$X$$ which we believe to be distributed uniformly on the interval $$[0, \theta]$$ for some unknown $$\theta > 0$$. In this statistical problem, we are implicitly considering the set $$\mathcal{P}$$ of Borel probability measures on $$\mathbb{R}$$ consisting of the uniform distributions on all intervals of the form $$[0, \theta]$$. That is, if $$\lambda$$ denotes Lebesgue measure and, for $$\theta > 0$$, $$P_\theta$$ denotes the $$\operatorname{Uniform}([0, \theta])$$ distribution (i.e., $$P_\theta(A) = \frac{1}{\theta} \lambda(A \cap [0, \theta]) = \int_A \frac{1}{\theta} \mathbf{1}_{[0, \theta]}(x) \, dx$$ for every Borel $$A \subseteq \mathbb{R}$$), then we simply have $$\mathcal{P} = \{P_\theta : \theta > 0\}.$$ This is the set of candidate distributions for our measurement $$X$$.

The family $$\mathcal{P}$$ is clearly dominated by Lebesgue measure $$\lambda$$ (which is $$\sigma$$-finite), so the lemma above (with $$\aleph = \mathcal{P}$$) guarantees the existence of a sequence $$\{c_i\}_{i=1}^\infty$$ of nonnegative numbers summing to $$1$$ and a sequence $$\{Q_i\}_{i=1}^\infty$$ of uniform distributions in $$\mathcal{P}$$ such that $$P_\theta \ll \sum_{i=1}^\infty c_i Q_i$$ for each $$\theta > 0$$. In this example, we can construct such sequences explicitly!

First, let $$(\theta_i)_{i=1}^\infty$$ be an enumeration of the positive rational numbers (this can be done explicitly), and let $$Q_i = P_{\theta_i}$$ for each $$i$$. Next, let $$c_i = 2^{-i}$$, so that $$\sum_{i=1}^\infty c_i = 1$$. I claim that this combination of $$\{c_i\}_{i=1}^\infty$$ and $$\{Q_i\}_{i=1}^\infty$$ works.

To see this, fix $$\theta > 0$$ and let $$A$$ be a Borel subset of $$\mathbb{R}$$ such that $$\sum_{i=1}^\infty c_i Q_i(A) = 0$$. We need to show that $$P_\theta(A) = 0$$. Since $$\sum_{i=1}^\infty c_i Q_i(A) = 0$$ and each summand is non-negative, it follows that $$c_i Q_i(A) = 0$$ for each $$i$$. Moreover, since each $$c_i$$ is positive, it follows that $$Q_i(A) = 0$$ for each $$i$$. That is, for all $$i$$ we have $$Q_i(A) = P_{\theta_i}(A) = \frac{1}{\theta_i} \lambda(A \cap [0, \theta_i]) = 0.$$ Since each $$\theta_i$$ is positive, it follows that $$\lambda(A \cap [0, \theta_i]) = 0$$ for each $$i$$.

Now choose a subsequence $$\{\theta_{i_k}\}_{k=1}^\infty$$ of $$\{\theta_i\}_{i=1}^\infty$$ which converges to $$\theta$$ from above (this can be done since $$\mathbb{Q}$$ is dense in $$\mathbb{R}$$). Then $$A \cap [0, \theta_{\theta_{i_k}}] \downarrow A \cap [0, \theta]$$ as $$k \to \infty$$, so by continuity of measure we conclude that $$\lambda(A \cap [0, \theta]) = \lim_{k \to \infty} \lambda(A \cap [0, \theta_{i_k}]) = 0,$$ and so $$P_\theta(A) = 0$$. This proves the claim.

Thus, in this example we were able to explicitly construct a countable convex combination of probability measures from our dominated family which still dominates the entire family. The Lemma above guarantees that this can be done for any dominated family (at least as long as the dominating measure is $$\sigma$$-finite).

# The Halmos-Savage Theorem

So now on to the Halmos-Savage Theorem (for which I will use slightly different notation than in the question due to personal preference). Given the Halmos-Savage Theorem, the Fisher-Neyman factorization theorem is just one application of the Doob-Dynkin lemma and the chain rule for Radon-Nikodym derivatives away!

Halmos-Savage Theorem. Let $$(\mathcal{X}, \mathcal{B}, \mathcal{P})$$ be a dominated statistical model (meaning that $$\mathcal{P}$$ is a set of probability measures on $$\mathcal{B}$$ and there is a $$\sigma$$-finite measure $$\mu$$ on $$\mathcal{B}$$ such that $$P \ll \mu$$ for all $$P \in \mathcal{P}$$). Let $$T : (\mathcal{X}, \mathcal{B}) \to (\mathcal{T}, \mathcal{C})$$ be a measurable function, where $$(T, \mathcal{C})$$ is a standard Borel space. Then the following are equivalent:

1. $$T$$ is sufficient for $$\mathcal{P}$$ (meaning that there is a probability kernel $$r : \mathcal{B} \times \mathcal{T} \to [0, 1]$$ such that $$r(B, T)$$ is a version of $$P(B \mid T)$$ for all $$B \in \mathcal{B}$$ and $$P \in \mathcal{P}$$).
2. There exists a sequence $$\{c_i\}_{i=1}^\infty$$ of nonnegative numbers such that $$\sum_{i=1}^\infty c_i = 1$$ and a sequence $$\{P_i\}_{i=1}^\infty$$ of probability measures in $$\mathcal{P}$$ such that $$P \ll P^*$$ for all $$P \in \mathcal{P}$$, where $$P^* = \sum_{i=1}^\infty c_i P_i$$, and for each $$P \in \mathcal{P}$$ there exists a $$T$$-measurable version of $$dP/dP^*$$.

Proof. By the lemma above, we may immediately replace $$\mu$$ by $$P^* = \sum_{i=1}^\infty c_i P_i$$ for some sequence $$\{c_i\}_{i=1}^\infty$$ of nonnegative numbers such that $$\sum_{i=1}^\infty c_i = 1$$ and a sequence $$\{P_i\}_{i=1}^\infty$$ of probability measures in $$\mathcal{P}$$.

(1. implies 2.) Suppose $$T$$ is sufficient. Then we must show that there are $$T$$-measurable versions of $$dP/dP^*$$ for all $$P \in \mathcal{P}$$. Let $$r$$ be the probability kernel in the statement of the theorem. For each $$A \in \sigma(T)$$ and $$B \in \mathcal{B}$$ we have \begin{aligned} P^*(A \cap B) &= \sum_{i=1}^\infty c_i P_i(A \cap B) \\ &= \sum_{i=1}^\infty c_i \int_A P_i(B \mid T) \, dP_i \\ &= \sum_{i=1}^\infty c_i \int_A r(B, T) \, dP_i \\ &= \int_A r(B, T) \, dP^*. \end{aligned} Thus $$r(B, T)$$ is a version of $$P^*(B \mid T)$$ for all $$B \in \mathcal{B}$$.

For each $$P \in \mathcal{P}$$, let $$f_P$$ denote a version of the Radon-Nikodym derivative $$dP/dP^*$$ on the measurable space $$(\mathcal{X}, \sigma(T))$$ (so in particular $$f_P$$ is $$T$$-measurable). Then for all $$B \in \mathcal{B}$$ and $$P \in \mathcal{P}$$ we have \begin{aligned} P(B) &= \int_{\mathcal{X}} P(B \mid T) \, dP \\ &= \int_{\mathcal{X}} r(B, T) \, dP \\ &= \int_{\mathcal{X}} r(B, T) f_P \, dP^* \\ &= \int_{\mathcal{X}} P^*(B \mid T) f_P \, dP^* \\ &= \int_{\mathcal{X}} E_{P^*}[\mathbf{1}_B f_P \mid T] \, dP^* \\ &= \int_B f_P \, dP^*. \end{aligned} Thus in fact $$f_P$$ is a $$T$$-measurable version of $$dP/dP^*$$ on $$(\mathcal{X}, \mathcal{B})$$. This proves that the first condition of the theorem implies the second.

(2. implies 1.) Suppose one can choose a $$T$$-measurable version $$f_P$$ of $$dP/dP^*$$ for each $$P \in \mathcal{P}$$. For each $$B \in \mathcal{B}$$, let $$r(B, t)$$ denote a particular version of $$P^*(B \mid T = t)$$ (e.g., $$r(B, t)$$ is a function such that $$r(B, T)$$ is a version of $$P^*(B \mid T)$$). Since $$(T, \mathcal{C})$$ is a standard Borel space, we may choose $$r$$ in a way that makes it a probability kernel (see, e.g., Theorem B.32 in Schervish's Theory of Statistics (1995)). We will show that $$r(B, T)$$ is a version of $$P(B \mid T)$$ for any $$P \in \mathcal{P}$$ and any $$B \in \mathcal{B}$$. Thus, let $$A \in \sigma(T)$$ and $$B \in \mathcal{B}$$ be given. Then for all $$P \in \mathcal{P}$$ we have \begin{aligned} P(A \cap B) &= \int_A \mathbf{1}_B f_P \, dP^* \\ &= \int_A E_{P^*}[\mathbf{1}_B f_P \mid T] \, dP^* \\ &= \int_A P^*(B \mid T) f_P \, dP^* \\ &= \int_A r(B, T) f_P \, dP^* \\ &= \int_A r(B, T) \, dP. \end{aligned} This shows that $$r(B, T)$$ is a version of $$P(B \mid T)$$ for any $$P \in \mathcal{P}$$ and any $$B \in \mathcal{B}$$, and the proof is done.

Summary. The important technical result underlying the Halmos-Savage theorem as presented here is the fact that a dominated family of probability measures is actually dominated by a countable convex combination of probability measures from that family. Given that result, the rest of the Halmos-Savage theorem is mostly just manipulations with basic properties of Radon-Nikodym derivatives and conditional expectations.