Pairwise independence of sufficient statistics in exponential families

Question

tl;dr: A simple argument seems to show that the sufficient statistics in a canonical exponential family are necessarily pairwise independent, which is clearly not true. So, what goes wrong in the derivation below?

Derivation (updated). This argument avoids reparametrization issues. Suppose we have an exponential family $\mathfrak{P}=\{p_{\theta}:\theta\in\Omega\}$ given by \begin{align} p_{\theta}(x) &= h(x)e^{\theta\cdot x - A(\theta)}, \quad x\in\mathbb{R}^{k}. \quad (*) \end{align}

Then obviously \begin{align*} p_{\theta}(x) &= h(x)e^{-A(\theta)}\prod_{i=1}^{k}e^{\theta_{i}x_{i}}. \end{align*}

We can assume $h(x)=1$ without any loss of generality (e.g. by absorbing $h(x)$ into the dominating measure). Reparametrizing in terms of the sufficient statistics $t$, We have the factorization $p_{\theta}(x)=f_{1}(x_{1})\cdots f_{k}(x_{k})$ (e.g. let $f_1(x_1)=e^{-A(\theta)}e^{\theta_{1}x_{1}}$ and $f_i(x_i)=e^{\theta_{i}x_{i}}$ for $i>1$). This factorization is enough to ensure that $x_{1},\ldots,x_{k}$ are independent.

Thus, the sufficient statistics in a canonical exponential family are always mutually independent. What goes wrong in this argument?

Note. There always exists a dominating measure such that $(*)$ holds, see Propsition~1.5 here. The same author calls this a standard exponential family. For the curious, this question arose from trying to understand some of the arguments in this textbook.

(Update: @Henry below has a very simple counterexample that proves that this is definitely false.)

Answer 1

The problem with this "paradox" comes from absorbing $h(x)$ into the dominating measure and then forgetting about the dominating measure.

The most common definition of a probability density is associated with a measure that is a product measure, like the Lebesgue measure. In this case, it is straightforward to prove that $p(x,y)=p_X(x)p_Y(y)$ implies that $X$ and $Y$ are independent: \begin{align*}\mathbb{P}(X\in A, Y\in B)&=\int_{A\times B} p_X(x)p_Y(y)\text{d}\lambda(x,y)\\&=\int_{A}\int_{B} p_X(x)p_Y(y)\text{d}\lambda(x)\text{d}\lambda(y)\\&=\mathbb{P}(X\in A)\mathbb{P}(Y\in B)\end{align*}

This result does not, however, extend to arbitrary dominating measures. The fact that a density (i.e. Radon-Nikodym derivative) against an arbitrary measure $\lambda$ factorises as $p(x,y)=p_X(x)p_Y(y)$ (with $p_x$ and $p_y$ integrable against the projected measures $\lambda_x$ and $\lambda_y$) does not make the components $X$ and $Y$ independent. It depends on the measure $\lambda$.

"Independence as a product" is achieved in terms of probabilities or measures, not in terms of densities.

A simple [counter-]example is made of the switch from the Lebesgue measure $\lambda_0$ to the new measure $$\exp\{\varrho xy\}\text{d}\lambda_0(x,y)$$ The distribution with density $$p(x,y)=\exp\left\{-\frac{x^2}{2}-\frac{y^2}{2}\right\}$$ against this measure is the Gaussian distribution with non-zero covariance $$-\frac{\varrho}{1-\varrho^2}.$$

An even simpler [counter-]example [or re-expression of the above] is to switch from the Lebesgue measure $\lambda_0$ to the new (Gaussian) measure $$\frac{(1-\varrho^2)^{1/2}}{2\pi}\exp\left\{-\frac{x^2}{2}-\frac{y^2}{2}+\varrho xy\right\}\text{d}\lambda_0(x,y)$$ and to consider the constant density $p(x,y)=1$ against this alternative dominating measure. We are again facing a distribution that is Gaussian with non-zero covariance but which has a product density.

Answer 2

I think it is not true

Consider a sample of $n$ observations from a normal distributed random variable $N(\mu, \sigma^2)$ with unknown mean and variance, which is from an exponential family

• $\left (\sum x_i, \sum x_i^2\right)$ is a sufficient statistic

• $\sum x_i$ and $\sum x_i^2$ are not independent