I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, we have:

$$ p_{\theta}(x) = \frac{1}{Z(\theta)}\exp\left(\theta^{T}\phi(x)\right) $$

where $$ Z(\theta) = \int\exp\left(\theta^{T}\phi(x)\right) dx $$

Now, to compute the moments or rather to show the moment generating property, there are steps to compute the log normalizer with respect to $\theta$.

So, I want to compute the second derivative of $\log Z(\theta)$. I did the following:

$$ \nabla\nabla\log Z(\theta) = -\frac{1}{Z(\theta)^2}\nabla\nabla Z(\theta) $$ Now, $$ \nabla Z(\theta) = \int\phi(x)\exp\left(\theta^T\phi(x)\right)dx $$ Similarly, $$ \nabla\nabla Z(\theta) = \int\phi(x)\phi(x)\exp\left(\theta^T\phi(x)\right)dx $$

So, $$ \nabla\nabla\log Z(\theta) = -\frac{1}{Z(\theta)}\int\frac{\phi(x)\phi(x)\exp\left(\theta^T\phi(x)\right)}{Z(\theta)}dx $$

which is:

$$ -\frac{1}{Z(\theta)} \mathbb{E}(x^2) $$

This is of course, the wrong result. It should be $\mathbb{E}(x^2) - \mathbb{E}^2(x)$.

I would greatly appreciate it if someone can show me where I have gone wrong.

  • $\begingroup$ I should learn how to take second derivatives…. Thanks for your help. $\endgroup$ – Luca Jan 22 '14 at 21:02
  • 1
    $\begingroup$ Indeed, it was just that. $\endgroup$ – Alecos Papadopoulos Jan 22 '14 at 23:00

To simplify notation, I will just use prime and double prime to denote first and second derivatives w.r.t to $\theta$, and write just $Z$. So

$$\frac {\partial ^2}{\partial \theta^2} \ln Z = \frac {{Z''Z -(Z')^2}}{Z^2}$$

We have

$$Z' = \int\phi(x)\exp\left(\theta^T\phi(x)\right)dx = Z \int p_{\theta}(x) \phi(x)dx $$


$$Z'' = \int[\phi(x)]^2\exp\left(\theta^T\phi(x)\right)dx = Z \int p_{\theta}(x) [\phi(x)]^2dx $$

Inserting into the basic relation the $Z$'s cancel off and so

$$\frac {\partial ^2}{\partial \theta^2} \ln Z = \int p_{\theta}(x) [\phi(x)]^2dx - \left(\int p_{\theta}(x) \phi(x)dx\right)^2$$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.