4
$\begingroup$

Given an exponential family distribution of the form

$$ f_X(x)=h(x)e^{\phi(\theta)^TT(x)-A(\theta)} $$

with natural parameter $\eta=\phi(\theta)$, sufficient statistic $T(x)$, and log partition function $A(\theta)$. We can absorb the $-A(\theta)$ term into the natural parameter vector by concatenating it:

$$ \phi'(\theta)=\left[\begin{array}{c}\phi(\theta)\\A(\theta)\end{array}\right],\quad T'(x)=\left[\begin{array}{c}T(x)\\-1\end{array}\right] $$

so that the log partition term disappears. We know that we can compute $\mathbb{E}[T'(x)]=\nabla_{\eta'}A'(\theta)$. Since $A'(\theta)=0$ this would mean that $\mathbb{E}[T'(x)]=[0,0,\cdots,0]^T$ for all distributions in the exponential family, which seems wrong. What is the problem here?

$\endgroup$
  • 2
    $\begingroup$ One thing I see is that the fact $\mathbb{E}[T(x)]=\nabla A(\theta)$ valid only for regular exponential families. That means set of valid parameters for $\phi(\theta)$ are in an open set. For the second parameterization I think we can't have an open set due to $A(\theta)$. Check Proposition 3.1 in Graphical Models, Exponential Families, and Variational Inference by Wainwright and Jordan (page 62). $\endgroup$ – user2939212 Oct 7 '17 at 5:03
  • $\begingroup$ @user2939212 Thanks, that's a useful hint. Could you elaborate a bit on how including $A(\theta)$ prevents $\phi'(\theta)$ from being an open set? $\endgroup$ – danijar Oct 7 '17 at 16:25
2
$\begingroup$

The definition of the log partition function, $$A(\theta) = \log \int h(x) \exp\{\phi(\theta)^T T(x) \}dx $$ makes it clear that given fixed functions $h(x)$ and $T(x)$, then $A(\theta)$ is completely determined by $\phi(\theta)$. In your second parameterization where you've absorbed the log-partition function to the natural parameter vector, the dimension of $\theta$ is one less than the dimension of $\phi'(\theta) = [\phi(\theta), -A(\theta)]$. Thus, the possible space of the natural parameter vector $\phi'(\theta)$ (the natural parameter space) is not an open set: it is a curve. As stated in the comment by @user2939212, the identity $\nabla A(\theta) = \mathbb{E}[T(x)]$ only holds if the natural parameter space is an open set (and so the family is regular).

Cases where the dimension of the parameter vector $\theta$ is less than the dimension of the natural parameter vector are called curved exponential families. For example, a Gaussian$(\mu, \sigma^2)$ where $\sigma = |\mu| = \theta$ is a curved exponential family: the natural parameter vector is $$\eta(\theta) = [\theta^{-1},-0.5\theta^{-2}].$$ The space spanned by $\eta(\theta)$ is a parabola, not an open set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.