# Can we absorb the partition function into the natural parameter vector?

Given an exponential family distribution of the form

$$f_X(x)=h(x)e^{\phi(\theta)^TT(x)-A(\theta)}$$

with natural parameter $\eta=\phi(\theta)$, sufficient statistic $T(x)$, and log partition function $A(\theta)$. We can absorb the $-A(\theta)$ term into the natural parameter vector by concatenating it:

$$\phi'(\theta)=\left[\begin{array}{c}\phi(\theta)\\A(\theta)\end{array}\right],\quad T'(x)=\left[\begin{array}{c}T(x)\\-1\end{array}\right]$$

so that the log partition term disappears. We know that we can compute $\mathbb{E}[T'(x)]=\nabla_{\eta'}A'(\theta)$. Since $A'(\theta)=0$ this would mean that $\mathbb{E}[T'(x)]=[0,0,\cdots,0]^T$ for all distributions in the exponential family, which seems wrong. What is the problem here?

• One thing I see is that the fact $\mathbb{E}[T(x)]=\nabla A(\theta)$ valid only for regular exponential families. That means set of valid parameters for $\phi(\theta)$ are in an open set. For the second parameterization I think we can't have an open set due to $A(\theta)$. Check Proposition 3.1 in Graphical Models, Exponential Families, and Variational Inference by Wainwright and Jordan (page 62). – user2939212 Oct 7 '17 at 5:03
• @user2939212 Thanks, that's a useful hint. Could you elaborate a bit on how including $A(\theta)$ prevents $\phi'(\theta)$ from being an open set? – danijar Oct 7 '17 at 16:25

The definition of the log partition function, $$A(\theta) = \log \int h(x) \exp\{\phi(\theta)^T T(x) \}dx$$ makes it clear that given fixed functions $h(x)$ and $T(x)$, then $A(\theta)$ is completely determined by $\phi(\theta)$. In your second parameterization where you've absorbed the log-partition function to the natural parameter vector, the dimension of $\theta$ is one less than the dimension of $\phi'(\theta) = [\phi(\theta), -A(\theta)]$. Thus, the possible space of the natural parameter vector $\phi'(\theta)$ (the natural parameter space) is not an open set: it is a curve. As stated in the comment by @user2939212, the identity $\nabla A(\theta) = \mathbb{E}[T(x)]$ only holds if the natural parameter space is an open set (and so the family is regular).
Cases where the dimension of the parameter vector $\theta$ is less than the dimension of the natural parameter vector are called curved exponential families. For example, a Gaussian$(\mu, \sigma^2)$ where $\sigma = |\mu| = \theta$ is a curved exponential family: the natural parameter vector is $$\eta(\theta) = [\theta^{-1},-0.5\theta^{-2}].$$ The space spanned by $\eta(\theta)$ is a parabola, not an open set.