Can we absorb the partition function into the natural parameter vector?

Question

Given an exponential family distribution of the form

$$ f_X(x)=h(x)e^{\phi(\theta)^TT(x)-A(\theta)} $$

with natural parameter $\eta=\phi(\theta)$, sufficient statistic $T(x)$, and log partition function $A(\theta)$. We can absorb the $-A(\theta)$ term into the natural parameter vector by concatenating it:

$$ \phi'(\theta)=\left[\begin{array}{c}\phi(\theta)\\A(\theta)\end{array}\right],\quad T'(x)=\left[\begin{array}{c}T(x)\\-1\end{array}\right] $$

so that the log partition term disappears. We know that we can compute $\mathbb{E}[T'(x)]=\nabla_{\eta'}A'(\theta)$. Since $A'(\theta)=0$ this would mean that $\mathbb{E}[T'(x)]=[0,0,\cdots,0]^T$ for all distributions in the exponential family, which seems wrong. What is the problem here?

Answer 1

The definition of the log partition function, $$A(\theta) = \log \int h(x) \exp\{\phi(\theta)^T T(x) \}dx $$ makes it clear that given fixed functions $h(x)$ and $T(x)$, then $A(\theta)$ is completely determined by $\phi(\theta)$. In your second parameterization where you've absorbed the log-partition function to the natural parameter vector, the dimension of $\theta$ is one less than the dimension of $\phi'(\theta) = [\phi(\theta), -A(\theta)]$. Thus, the possible space of the natural parameter vector $\phi'(\theta)$ (the natural parameter space) is not an open set: it is a curve. As stated in the comment by @user2939212, the identity $\nabla A(\theta) = \mathbb{E}[T(x)]$ only holds if the natural parameter space is an open set (and so the family is regular).

Cases where the dimension of the parameter vector $\theta$ is less than the dimension of the natural parameter vector are called curved exponential families. For example, a Gaussian$(\mu, \sigma^2)$ where $\sigma = |\mu| = \theta$ is a curved exponential family: the natural parameter vector is $$\eta(\theta) = [\theta^{-1},-0.5\theta^{-2}].$$ The space spanned by $\eta(\theta)$ is a parabola, not an open set.