KL divergence minimisation equation

Question

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, we have:

$$ p_{\theta}(x) = \frac{1}{Z(\theta)}\exp\left(\theta^{T}\phi(x)\right) $$

where $$ Z(\theta) = \int\exp\left(\theta^{T}\phi(x)\right) dx $$

Now, to compute the moments or rather to show the moment generating property, there are steps to compute the log normalizer with respect to $\theta$.

So, I want to compute the second derivative of $\log Z(\theta)$. I did the following:

$$ \nabla\nabla\log Z(\theta) = -\frac{1}{Z(\theta)^2}\nabla\nabla Z(\theta) $$ Now, $$ \nabla Z(\theta) = \int\phi(x)\exp\left(\theta^T\phi(x)\right)dx $$ Similarly, $$ \nabla\nabla Z(\theta) = \int\phi(x)\phi(x)\exp\left(\theta^T\phi(x)\right)dx $$

So, $$ \nabla\nabla\log Z(\theta) = -\frac{1}{Z(\theta)}\int\frac{\phi(x)\phi(x)\exp\left(\theta^T\phi(x)\right)}{Z(\theta)}dx $$

which is:

$$ -\frac{1}{Z(\theta)} \mathbb{E}(x^2) $$

This is of course, the wrong result. It should be $\mathbb{E}(x^2) - \mathbb{E}^2(x)$.

I would greatly appreciate it if someone can show me where I have gone wrong.

Answer 1

To simplify notation, I will just use prime and double prime to denote first and second derivatives w.r.t to $\theta$, and write just $Z$. So

$$\frac {\partial ^2}{\partial \theta^2} \ln Z = \frac {{Z''Z -(Z')^2}}{Z^2}$$

We have

$$Z' = \int\phi(x)\exp\left(\theta^T\phi(x)\right)dx = Z \int p_{\theta}(x) \phi(x)dx $$

and

$$Z'' = \int[\phi(x)]^2\exp\left(\theta^T\phi(x)\right)dx = Z \int p_{\theta}(x) [\phi(x)]^2dx $$

Inserting into the basic relation the $Z$'s cancel off and so

$$\frac {\partial ^2}{\partial \theta^2} \ln Z = \int p_{\theta}(x) [\phi(x)]^2dx - \left(\int p_{\theta}(x) \phi(x)dx\right)^2$$