# KL divergence minimisation equation

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, we have:

$$p_{\theta}(x) = \frac{1}{Z(\theta)}\exp\left(\theta^{T}\phi(x)\right)$$

where $$Z(\theta) = \int\exp\left(\theta^{T}\phi(x)\right) dx$$

Now, to compute the moments or rather to show the moment generating property, there are steps to compute the log normalizer with respect to $\theta$.

So, I want to compute the second derivative of $\log Z(\theta)$. I did the following:

$$\nabla\nabla\log Z(\theta) = -\frac{1}{Z(\theta)^2}\nabla\nabla Z(\theta)$$ Now, $$\nabla Z(\theta) = \int\phi(x)\exp\left(\theta^T\phi(x)\right)dx$$ Similarly, $$\nabla\nabla Z(\theta) = \int\phi(x)\phi(x)\exp\left(\theta^T\phi(x)\right)dx$$

So, $$\nabla\nabla\log Z(\theta) = -\frac{1}{Z(\theta)}\int\frac{\phi(x)\phi(x)\exp\left(\theta^T\phi(x)\right)}{Z(\theta)}dx$$

which is:

$$-\frac{1}{Z(\theta)} \mathbb{E}(x^2)$$

This is of course, the wrong result. It should be $\mathbb{E}(x^2) - \mathbb{E}^2(x)$.

I would greatly appreciate it if someone can show me where I have gone wrong.

• I should learn how to take second derivatives…. Thanks for your help. – Luca Jan 22 '14 at 21:02
• Indeed, it was just that. – Alecos Papadopoulos Jan 22 '14 at 23:00

To simplify notation, I will just use prime and double prime to denote first and second derivatives w.r.t to $\theta$, and write just $Z$. So

$$\frac {\partial ^2}{\partial \theta^2} \ln Z = \frac {{Z''Z -(Z')^2}}{Z^2}$$

We have

$$Z' = \int\phi(x)\exp\left(\theta^T\phi(x)\right)dx = Z \int p_{\theta}(x) \phi(x)dx$$

and

$$Z'' = \int[\phi(x)]^2\exp\left(\theta^T\phi(x)\right)dx = Z \int p_{\theta}(x) [\phi(x)]^2dx$$

Inserting into the basic relation the $Z$'s cancel off and so

$$\frac {\partial ^2}{\partial \theta^2} \ln Z = \int p_{\theta}(x) [\phi(x)]^2dx - \left(\int p_{\theta}(x) \phi(x)dx\right)^2$$