tl;dr: A simple argument seems to show that the sufficient statistics in a canonical exponential family are necessarily pairwise independent, which is clearly not true. So, what goes wrong in the derivation below?

Derivation (updated). This argument avoids reparametrization issues. Suppose we have an exponential family $\mathfrak{P}=\{p_{\theta}:\theta\in\Omega\}$ given by \begin{align} p_{\theta}(x) &= h(x)e^{\theta\cdot x - A(\theta)}, \quad x\in\mathbb{R}^{k}. \quad (*) \end{align}

Then obviously \begin{align*} p_{\theta}(x) &= h(x)e^{-A(\theta)}\prod_{i=1}^{k}e^{\theta_{i}x_{i}}. \end{align*}

We can assume $h(x)=1$ without any loss of generality (e.g. by absorbing $h(x)$ into the dominating measure). Reparametrizing in terms of the sufficient statistics $t$, We have the factorization $p_{\theta}(x)=f_{1}(x_{1})\cdots f_{k}(x_{k})$ (e.g. let $f_1(x_1)=e^{-A(\theta)}e^{\theta_{1}x_{1}}$ and $f_i(x_i)=e^{\theta_{i}x_{i}}$ for $i>1$). This factorization is enough to ensure that $x_{1},\ldots,x_{k}$ are independent.

Thus, the sufficient statistics in a canonical exponential family are always mutually independent. What goes wrong in this argument?

Note. There always exists a dominating measure such that $(*)$ holds, see Propsition~1.5 here. The same author calls this a standard exponential family. For the curious, this question arose from trying to understand some of the arguments in this textbook.

(Update: @Henry below has a very simple counterexample that proves that this is definitely false.)

  • 3
    Exactly what theorem are you quoting where you write "this is enough to ensure ... independent"? Although you appear to have in mind one about factorization of a joint density, I don't see the Jacobian of the transformation from $x$ to $(t_1,\ldots, t_k)$ anywhere. – whuber Aug 28 '17 at 19:31
  • The $x$ themselves are the same across the $f_i(t)$, hence you need not have independence between the $t_i(x)$, which are just different functions of the same underlying $x$. – jbowman Aug 28 '17 at 19:37
  • @whuber: That is indeed a flaw, thanks for pointing it out! Even so, there is a way around this, and independence still seems to persist. See my modified argument above. Regarding the theorem I am using: Not really a theorem, but once we have the factorization of a joint into the product $f_1\cdots f_k$, this implies independence (easy calculation, or see Dawid, 1979). – JohnA Aug 28 '17 at 22:41
  • 1
    In your reformulation, Henry's answer no longer is a counterexample. I will continue suggesting that you are playing fast and loose with the notation and with your application of this factorization. I think if you were to write things down carefully and consistently, the error would be prominently revealed. – whuber Aug 29 '17 at 14:54
  • 1
    @Xi'an Has pointed out one place where your notation is problematic: by making $h$ disappear, you created the paradox. – whuber Aug 30 '17 at 15:03
up vote 4 down vote accepted

The problem with this "paradox" comes from absorbing $h(x)$ into the dominating measure and then forgetting about the dominating measure.

The most common definition of a probability density is associated with a measure that is a product measure, like the Lebesgue measure. In this case, it is straightforward to prove that $p(x,y)=p_X(x)p_Y(y)$ implies that $X$ and $Y$ are independent: \begin{align*}\mathbb{P}(X\in A, Y\in B)&=\int_{A\times B} p_X(x)p_Y(y)\text{d}\lambda(x,y)\\&=\int_{A}\int_{B} p_X(x)p_Y(y)\text{d}\lambda(x)\text{d}\lambda(y)\\&=\mathbb{P}(X\in A)\mathbb{P}(Y\in B)\end{align*}

This result does not, however, extend to arbitrary dominating measures. The fact that a density (i.e. Radon-Nikodym derivative) against an arbitrary measure $\lambda$ factorises as $p(x,y)=p_X(x)p_Y(y)$ (with $p_x$ and $p_y$ integrable against the projected measures $\lambda_x$ and $\lambda_y$) does not make the components $X$ and $Y$ independent. It depends on the measure $\lambda$.

"Independence as a product" is achieved in terms of probabilities or measures, not in terms of densities.

A simple [counter-]example is made of the switch from the Lebesgue measure $\lambda_0$ to the new measure $$\exp\{\varrho xy\}\text{d}\lambda_0(x,y)$$ The distribution with density $$p(x,y)=\exp\left\{-\frac{x^2}{2}-\frac{y^2}{2}\right\}$$ against this measure is the Gaussian distribution with non-zero covariance $$-\frac{\varrho}{1-\varrho^2}.$$

An even simpler [counter-]example [or re-expression of the above] is to switch from the Lebesgue measure $\lambda_0$ to the new (Gaussian) measure $$\frac{(1-\varrho^2)^{1/2}}{2\pi}\exp\left\{-\frac{x^2}{2}-\frac{y^2}{2}+\varrho xy\right\}\text{d}\lambda_0(x,y)$$ and to consider the constant density $p(x,y)=1$ against this alternative dominating measure. We are again facing a distribution that is Gaussian with non-zero covariance but which has a product density.

  • In your example, neither $\lambda_0$ nor the new measure are probability measures. The same problem arises in my derivation by absorbing $h(x)$ into the dominating measure. So to clarify: These counterexamples arise precisely because the underlying measure is not a probability measure. Or are there also counterexamples where the dominating measure is also a probability measure? – JohnA Aug 30 '17 at 17:11
  • @JohnAustin: I do not understand the objection about the dominating measure not being a probability measure. This is not a counter-example, this is an explanation of why the notion of independence is impacted by both the density and the measure. – Xi'an Aug 30 '17 at 18:20
  • I believe I see the issue now. Please see my edits to your response for a clarification. – JohnA Aug 30 '17 at 20:07

I think it is not true

Consider a sample of $n$ observations from a normal distributed random variable $N(\mu, \sigma^2)$ with unknown mean and variance, which is from an exponential family

  • $\left (\sum x_i, \sum x_i^2\right)$ is a sufficient statistic

  • $\sum x_i$ and $\sum x_i^2$ are not independent

  • I think the question was about the canonical sufficient statistics – kjetil b halvorsen Aug 28 '17 at 12:28
  • @kjetilbhalvorsen - if the canonical form of the likelihood is $\left(\frac1{\sqrt{2\pi}} \right)^n \exp\left( \frac{\mu}{\sigma^2}\sum x_i-\frac{1}{2\sigma^2}\sum x_i^2 - \frac{\mu^2}{2\sigma^2} -\frac{\log(\sigma^2)}{2}\right)$ then I would have thought the canonical parameters would be $\frac{\mu}{\sigma^2}$ and $-\frac{1}{2\sigma^2}$ and the corresponding sufficient statistics would be $\left (\sum x_i, \sum x_i^2\right)$. But perhaps I have misunderstood – Henry Aug 28 '17 at 12:56
  • This looks correct to me. Such a simple counterexample, too. The question then remains: What is wrong with the derivation I gave above? – JohnA Aug 28 '17 at 13:09

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.