Suppose $\boldsymbol X$ belongs to the exponential family,

$$ f_X\!\left(\,\mathbf{x} ; \boldsymbol \theta\,\right) = h(\mathbf{x}) \, \exp\!\Big(\,\boldsymbol\eta({\boldsymbol \theta}) \cdot \mathbf{T}(\mathbf{x}) - A({\boldsymbol \theta})\,\Big)$$

with $\mathbf{T}(\mathbf{x})$ the sufficient statistic.

Is there a way to factorize $f_X\!\left(\,\mathbf{x};\boldsymbol \theta\,\right)$ such that

$$f_X\!\left(\,\mathbf{x};\boldsymbol \theta\,\right) = f_T\!\left(\,\mathbf{T};\boldsymbol \theta\,\right) f_{X|T}\!\left(\,\mathbf{x} | \mathbf{T}\right)$$

where $f_{X|T}\!\left(\,\mathbf{x} | \mathbf{T}\right)$ is the distribution of $\boldsymbol X$ conditional on $\mathbf{T}$, and $f_T\!\left(\,\mathbf{T};\boldsymbol \theta\,\right)$ is the marginal distribution of $\mathbf{T}$?


2 Answers 2


In a discrete setting, when the sampling space $\mathfrak X^n$ (assuming an iid sample of size $n$ from an exponential family) is finite or countable, the conditional distribution of $$\mathbb P(\mathbf X_n=\mathbf x_n;\theta\vert T_n(\mathbf X_n)=t_n)= \dfrac{\mathbb P(\mathbf X_n=\mathbf x_n;\theta)}{\mathbb P(T_n(\mathbf X_n)=t_n;\theta)}$$ Hence, using the counting measures on $\mathfrak X^n$ and $T(\mathfrak X^n)$, respectively, $$f(\mathbf x_n;\theta) = f(\mathbf x_n\vert T_n(\mathbf X_n)=t_n) \times f(t_n;\theta)$$ In the continuous case, the result is somehow a tautology since the choice of the (three) dominating measures is open. To start, we can wlog assume that$$T_n(\mathbf x_n)=\sum_{i=1}^n x_i$$ and that the density with respect to a measure $\lambda^{\otimes n}$ on $\mathfrak X^n$ is $$f(\mathbf x_n;\theta) = \prod_{i=1}^n h(x_i) \times e^{\theta\cdot T_n(\mathbf x_n) - n A(\theta)}$$ Changing the measure into$$\text d\mu(x) \stackrel{\text{def}}{=} h(x)\,\text d\mu(x)$$leads to the density with respect to the measure $\mu^{\otimes n}$ on $\mathfrak X^n$ $$f(\mathbf x_n;\theta) = e^{\theta\cdot T_n(\mathbf x_n) - n A(\theta)}$$ The joint density of $(T_n(\mathbf X),X_2,\ldots,X_n)$ wrt the measure $\mu_{T_n}\otimes\mu^{\otimes n-1}$ on $T_n(\mathfrak X^n)\times\mathfrak X^{n-1}$ is then $$f(t_n,x_2,\ldots,x_n;\theta) = e^{\theta\cdot t_n - n A(\theta)}\tag{1}$$ since the Jacobian is equal to one. The marginal density of $T_n(\mathbf X)$ wrt $\mu_{T_n}$ is thus $$f(t_n;\theta) = e^{\theta\cdot t_n - n A(\theta)}\times \tilde h(t_n)$$ where the last term $\tilde h(t_n)$ is the mass of $$T_n(t_n-\sum_{i\ge 2} x_i,x_2,\ldots,x_n)=t_n\tag{2}$$ under $\mu^{\otimes n-1}$. Changing the measure $\mu_{T_n}$ to the new measure $\xi_{T_n}$ defined by $$\text d\xi_{T_n} (t)= \tilde h(t)\text d\mu_{T_n}(t)$$ leads to the following density of $T_n(\mathbf X)$ wrt $\xi_{T_n}$: $$f(t_n;\theta) = e^{\theta\cdot t_n - n A(\theta)}$$ If one defines $\zeta_{t_n}$ as the mass one uniform measure over the slice $$\left\{(x_1,x_2,\ldots,x_n); \ T_n(x_1,x_2,\ldots,x_n)=\sum_{i=1}^n x_i=t_n\right\},$$ the conditional density of $\mathbf X$ conditional on $T_n(\mathbf X)$ is one and $$f_X\left(\,\mathbf{x}_n;\theta\,\right) = f_{T_n}\!\left(\,t_n; \theta\,\right) f_{X|T=t_n}\!\left(\,\mathbf{x}_n | T_n=t_n\right)$$ when the densities are taken wrt $\mu^{\otimes n-1}$, $\xi_{T_n}$, and $\zeta_{t_n}$, respectively.

(Note: Using $f$ for different densities is an abuse of notation, but avoids an explosion in said notation.)


To get $f_{T}\!\left(\,\mathbf{T} ; \boldsymbol\theta\right)$, we take the full density and integrate $\mathbf{x}$ out:

\begin{align}f_{T}\!\left(\,\mathbf{T} ; \boldsymbol\theta\right) &= \int_{{\bf x} : {\bf T(x)} = {\bf T}} f_X\!\left(\,\mathbf{x} ; \boldsymbol \theta\,\right) d {\bf x} \\ &= \exp\!\Big(\,\boldsymbol\eta({\boldsymbol \theta}) \ \mathbf{T} - A({\boldsymbol \theta})\,\Big) \int_{{\bf x} : {\bf T(x)} = {\bf T}} h(\mathbf{x}) d {\bf x} \\ &= \exp\!\Big(\,\boldsymbol\eta({\boldsymbol \theta}) \ \mathbf{T} - A({\boldsymbol \theta})\,\Big) h_T({\bf T}) \end{align} Where $$h_T({\bf T}) = \int_{{\bf x} : {\bf T(x)} = {\bf T}} h(\mathbf{x}) d {\bf x}$$

Consequentially, we have

$$ f_{X|T}\!\left(\,\mathbf{x} | \mathbf{T}\right) = \frac{ h({\bf x}) }{h_T({\bf T}({\bf x}))} $$

  • 1
    $\begingroup$ The integral$$\int_{{\bf x} : {\bf T(x)} = {\bf T}} f_X\!\left(\,\mathbf{x} ; \boldsymbol \theta\,\right) \text d {\bf x}$$is not defined when $\text dx$ denotes the Lebesgue measure. $\endgroup$
    – Xi'an
    Commented Dec 9, 2020 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.