Minimizing Kullback-Leibler divergence

Question

This paper is showing how KL divergence can be minimized by matching the expected values of the sufficient statistics. More precisely, For any distribution p of the exponential family with pdf: $$ p_{\theta}(x) = \frac{1} {Z(\theta)}\exp(\theta^T \phi(x))$$ the distribution $$p_{θ∗} $$ which minimises the Kullback-Leibler divergence, $$ KL (p||p_{θ∗})$$ over the exponential family with natural statistic $\phi$ is implicitly given by: $$E_{p_{\theta}*(x)} [\phi(x)] = E_{p_{(x)}} [\phi(x)]$$

I need help to show that this is true for the particular case of a normal-gamma distribution. So I need to make the same proof that is on the paper but for the special case of Normal gamma.

Answer 1

We already have that the sufficient statistics are $$ T_1 = \ln T,\; T_2 = T,\; T_3 = TX,\; T_4=TX^2. $$

You should select the approximating density $q(X,T)$. Then, by Theorem 1, you simply equate the true expectations with the expectations under the approximating distribution. That is, $$ \begin{align} E_q(\ln T) &= \psi(\alpha) - \ln(\beta)\\ E_q(T) &= \frac{\alpha}{\beta},\\ E_q(TX) &= \frac{\alpha\mu}{\beta},\\ E_q(TX^2) &= \frac{\alpha\mu^2}{\beta} + \frac{1}{\lambda}. \end{align} $$ Until you specify what your approximating density is, this is all I can say for this problem. The cited note is explaining assumed density filtering where the posterior distribution is the true distribution and the approximating density $q$ is an exponential family.