Kullback Leibler Divergence between two Normal Whishart Distributions I'm having trouble to compute the KL Divergence between two normal-Wishart distributions. KL divergence from $Q$ to $P$ is defined as:
$$D_{\mathrm{KL}}(P \Vert Q) = \int p(x) \log \frac{p(x)}{q(x)}dx$$
and a normal-Wishart distribution is defined as:
$$f(\pmb{\mu},\pmb{\Lambda} \vert \pmb{\mu}_0, \lambda, \mathbf{W}, \nu) = \mathcal{N} \left(\pmb{\mu} \vert \pmb{\mu}_0, (\lambda \pmb{\Lambda})^{-1} \right) \mathcal{W}(\pmb{\Lambda} \vert \mathbf{W}, \nu) $$
where $\pmb{\mu}_0 \in \mathbb{R}^D, \mathbf{W} \in \mathbb{R}^{D\times D},\nu > D - 1,\lambda > 0$ are the parameters of the distribution. So we have:
$$D_{\mathrm{KL}} (f_0 \Vert f_1) = \int f_0(x) \log \frac{f_0(x)}{f_1(x)}dx$$
$$D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \log \frac{\mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda})}d\pmb{\mu}d\pmb{\Lambda}$$
because $\mathcal{N}$ and $\mathcal{W}$ are independent we have:
$$ D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] \\
= \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \left(\log \frac{\mathcal{N}_{0}(\pmb{\mu})}{\mathcal{N}_{1}(\pmb{\mu})} + \log \frac{\mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{W}_{1}(\pmb{\Lambda})}\right) d\pmb{\mu} d\pmb{\Lambda} $$
how can i proceed from here?
 A: You are getting closed. To ease the derivation, let's redefine some notations:
\begin{cases}
p(\pmb{\mu}, \pmb{\Lambda}) = p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{p}, (\lambda_{p} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{p}, \nu_{p} \right)\\
q(\pmb{\mu}, \pmb{\Lambda}) = q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{q}, (\lambda_{q} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{q}, \nu_{q} \right)
\end{cases}
The KL divergence of interest is:
\begin{align}
& D_{\mathrm{KL}} \left[ p(\pmb{\mu}, \pmb{\Lambda}) \Vert q(\pmb{\mu}, \pmb{\Lambda}) \right] \\
& = \int_{\mu} \int_{\Lambda} p(\pmb{\mu}, \pmb{\Lambda}) \ln \frac{p(\pmb{\mu}, \pmb{\Lambda})}{q(\pmb{\mu}, \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\
& = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\
& = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} + \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda}\\
& = \int_{\Lambda} p(\pmb{\Lambda}) \left[\int_{\mu} p(\pmb{\mu} \vert \pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} \right] d\pmb{\Lambda} + \int_{\Lambda} p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\Lambda}\\
& = \mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] + D_{\mathrm{KL}} \left[ p(\pmb{\Lambda}) \Vert q(\pmb{\Lambda}) \right].\\
& \tag{eq:KL_normal_wishart}
\label{eq:KL_normal_wishart}
\end{align}
To what I am aware of, there is a closed-form for the KL divergence between two Wishart distributions, corresponding to the second term. However, the first term is complicated, and I believe that further assumptions (eg. diagonal normal distributions) should be made to have a closed-form solution.
The first term is an expectation of the KL divergence between two normal distributions w.r.t. $p(\pmb{\Lambda})$, and also has a closed-form solution. To be specific, the KL divergence between 2 normal distributions can be written as:
\begin{aligned}[b]
& D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \\
& = \frac{1}{2} \left[ \mathrm{tr}\left( \lambda_{q} \pmb{\Lambda} \lambda_{p}^{-1} \pmb{\Lambda}^{-1} \right) + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + \ln \frac{\mathrm{det}(\lambda_{p} \pmb{\Lambda})}{\mathrm{det}(\lambda_{q} \pmb{\Lambda})}\right] \\
& = \frac{1}{2} \left[ D \frac{\lambda_{q}}{\lambda_{p}} + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + D \ln \frac{\lambda_{p}}{\lambda_{q}} \right].
\end{aligned}
Note that for a Wishart distribution: $\mathbb{E}_{p(\pmb{\Lambda})} \left[ \pmb{\Lambda} \right] = \nu_{p} \mathbf{W}_{p}$. Hence, the first term of $\eqref{eq:KL_normal_wishart}$ can be obtained as:
\begin{aligned}[b]
&\mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] \\
& = \frac{\lambda_{q}}{2} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \nu_{p} \mathbf{W}_{p} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) + \frac{D}{2} \left( \frac{\lambda_{q}}{\lambda_{p}} - \ln \frac{\lambda_{q}}{\lambda_{p}} - 1 \right).
\end{aligned}
