# Kullback Leibler Divergence between two Normal Whishart Distributions

I'm having trouble to compute the KL Divergence between two normal-Wishart distributions. KL divergence from $$Q$$ to $$P$$ is defined as:

$$D_{\mathrm{KL}}(P \Vert Q) = \int p(x) \log \frac{p(x)}{q(x)}dx$$

and a normal-Wishart distribution is defined as:

$$f(\pmb{\mu},\pmb{\Lambda} \vert \pmb{\mu}_0, \lambda, \mathbf{W}, \nu) = \mathcal{N} \left(\pmb{\mu} \vert \pmb{\mu}_0, (\lambda \pmb{\Lambda})^{-1} \right) \mathcal{W}(\pmb{\Lambda} \vert \mathbf{W}, \nu)$$

where $$\pmb{\mu}_0 \in \mathbb{R}^D, \mathbf{W} \in \mathbb{R}^{D\times D},\nu > D - 1,\lambda > 0$$ are the parameters of the distribution. So we have:

$$D_{\mathrm{KL}} (f_0 \Vert f_1) = \int f_0(x) \log \frac{f_0(x)}{f_1(x)}dx$$

$$D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \log \frac{\mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda})}d\pmb{\mu}d\pmb{\Lambda}$$

because $$\mathcal{N}$$ and $$\mathcal{W}$$ are independent we have:

$$D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] \\ = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \left(\log \frac{\mathcal{N}_{0}(\pmb{\mu})}{\mathcal{N}_{1}(\pmb{\mu})} + \log \frac{\mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{W}_{1}(\pmb{\Lambda})}\right) d\pmb{\mu} d\pmb{\Lambda}$$

how can i proceed from here?

You are getting closed. To ease the derivation, let's redefine some notations:

$$\begin{cases} p(\pmb{\mu}, \pmb{\Lambda}) = p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{p}, (\lambda_{p} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{p}, \nu_{p} \right)\\ q(\pmb{\mu}, \pmb{\Lambda}) = q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{q}, (\lambda_{q} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{q}, \nu_{q} \right) \end{cases}$$

The KL divergence of interest is:

\begin{align} & D_{\mathrm{KL}} \left[ p(\pmb{\mu}, \pmb{\Lambda}) \Vert q(\pmb{\mu}, \pmb{\Lambda}) \right] \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu}, \pmb{\Lambda}) \ln \frac{p(\pmb{\mu}, \pmb{\Lambda})}{q(\pmb{\mu}, \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} + \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda}\\ & = \int_{\Lambda} p(\pmb{\Lambda}) \left[\int_{\mu} p(\pmb{\mu} \vert \pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} \right] d\pmb{\Lambda} + \int_{\Lambda} p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\Lambda}\\ & = \mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] + D_{\mathrm{KL}} \left[ p(\pmb{\Lambda}) \Vert q(\pmb{\Lambda}) \right].\\ & \tag{eq:KL_normal_wishart} \label{eq:KL_normal_wishart} \end{align}

To what I am aware of, there is a closed-form for the KL divergence between two Wishart distributions, corresponding to the second term. However, the first term is complicated, and I believe that further assumptions (eg. diagonal normal distributions) should be made to have a closed-form solution.

The first term is an expectation of the KL divergence between two normal distributions w.r.t. $$p(\pmb{\Lambda})$$, and also has a closed-form solution. To be specific, the KL divergence between 2 normal distributions can be written as:

\begin{aligned}[b] & D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \\ & = \frac{1}{2} \left[ \mathrm{tr}\left( \lambda_{q} \pmb{\Lambda} \lambda_{p}^{-1} \pmb{\Lambda}^{-1} \right) + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + \ln \frac{\mathrm{det}(\lambda_{p} \pmb{\Lambda})}{\mathrm{det}(\lambda_{q} \pmb{\Lambda})}\right] \\ & = \frac{1}{2} \left[ D \frac{\lambda_{q}}{\lambda_{p}} + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + D \ln \frac{\lambda_{p}}{\lambda_{q}} \right]. \end{aligned}

Note that for a Wishart distribution: $$\mathbb{E}_{p(\pmb{\Lambda})} \left[ \pmb{\Lambda} \right] = \nu_{p} \mathbf{W}_{p}$$. Hence, the first term of $$\eqref{eq:KL_normal_wishart}$$ can be obtained as:

\begin{aligned}[b] &\mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] \\ & = \frac{\lambda_{q}}{2} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \nu_{p} \mathbf{W}_{p} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) + \frac{D}{2} \left( \frac{\lambda_{q}}{\lambda_{p}} - \ln \frac{\lambda_{q}}{\lambda_{p}} - 1 \right). \end{aligned}

• What exactly is a diagonal normal distribution? – SOULed_Outt Dec 17 '19 at 3:46
• It is a multivariate normal distribution with diagonal covariance matrix. – Cuong Dec 17 '19 at 3:50
• thanks! i will take a closer look and get back to you. – hh32 Dec 17 '19 at 17:43
• I have a question regarding the DKL part of the two normal distributions. How do you get from $\ln \frac{det(\lambda_p\Lambda)}{det(\lambda_q\Lambda)}$ to $D\ln\frac{\lambda_p}{\lambda_q}$? Also, shouldn't it be $\ln \frac{det(\lambda_q\Lambda)}{det(\lambda_p\Lambda)}$? – hh32 Dec 20 '19 at 9:58
• The covariance matrix in this case is: $(\lambda \pmb{\Lambda})^{-1}$. Hence, following the closed-form for KL divergence between 2 normal distributions, it should be something with p/q due to the inverse. Then, applying the property of matrix inverse: $\mathrm{det}(\alpha \mathbf{A}) = \alpha^{d} \mathrm{det}(\mathbf{A})$, where $d$ is the dimension of matrix $\mathbf{A}$. – Cuong Dec 20 '19 at 23:17