# Minimizing Kullback-Leibler divergence

This paper is showing how KL divergence can be minimized by matching the expected values of the sufficient statistics. More precisely, For any distribution p of the exponential family with pdf: $$p_{\theta}(x) = \frac{1} {Z(\theta)}\exp(\theta^T \phi(x))$$ the distribution $$p_{θ∗}$$ which minimises the Kullback-Leibler divergence, $$KL (p||p_{θ∗})$$ over the exponential family with natural statistic $$\phi$$ is implicitly given by: $$E_{p_{\theta}*(x)} [\phi(x)] = E_{p_{(x)}} [\phi(x)]$$

I need help to show that this is true for the particular case of a normal-gamma distribution. So I need to make the same proof that is on the paper but for the special case of Normal gamma.

We already have that the sufficient statistics are $$T_1 = \ln T,\; T_2 = T,\; T_3 = TX,\; T_4=TX^2.$$

You should select the approximating density $$q(X,T)$$. Then, by Theorem 1, you simply equate the true expectations with the expectations under the approximating distribution. That is, \begin{align} E_q(\ln T) &= \psi(\alpha) - \ln(\beta)\\ E_q(T) &= \frac{\alpha}{\beta},\\ E_q(TX) &= \frac{\alpha\mu}{\beta},\\ E_q(TX^2) &= \frac{\alpha\mu^2}{\beta} + \frac{1}{\lambda}. \end{align} Until you specify what your approximating density is, this is all I can say for this problem. The cited note is explaining assumed density filtering where the posterior distribution is the true distribution and the approximating density $$q$$ is an exponential family.

• I don't totally understand what you explained but the density that I want to approximate is another normalgamma
– sam
Mar 15, 2022 at 17:37
• That would be trivial. The KL divergence will obviously be minimized when the parameters are identical between the two normal-gamma distributions. Mar 15, 2022 at 17:48
• So here P and P* are normal gamma but they don't have the same parameters
– sam
Mar 16, 2022 at 13:14
• @SamiHadouaj They will once you’ve minimized the KL divergence. Mar 16, 2022 at 13:39
• @SamiHadouaj It is impossible to "find" a distribution that minimizes the KL divergence because the class of distributions is infinite. A family of distributions must be specified and that will give you a handle on the minimization step. And it can't be another normal gamma because the KL divergence is minimized when they're the same distribution, i.e., same parameters. Mar 16, 2022 at 19:29