I'm having trouble to compute the KL Divergence between two normal-Wishart distributions. KL divergence from $Q$ to $P$ is defined as:

$$D_{\mathrm{KL}}(P \Vert Q) = \int p(x) \log \frac{p(x)}{q(x)}dx$$

and a normal-Wishart distribution is defined as:

$$f(\pmb{\mu},\pmb{\Lambda} \vert \pmb{\mu}_0, \lambda, \mathbf{W}, \nu) = \mathcal{N} \left(\pmb{\mu} \vert \pmb{\mu}_0, (\lambda \pmb{\Lambda})^{-1} \right) \mathcal{W}(\pmb{\Lambda} \vert \mathbf{W}, \nu) $$

where $\pmb{\mu}_0 \in \mathbb{R}^D, \mathbf{W} \in \mathbb{R}^{D\times D},\nu > D - 1,\lambda > 0$ are the parameters of the distribution. So we have:

$$D_{\mathrm{KL}} (f_0 \Vert f_1) = \int f_0(x) \log \frac{f_0(x)}{f_1(x)}dx$$

$$D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \log \frac{\mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda})}d\pmb{\mu}d\pmb{\Lambda}$$

because $\mathcal{N}$ and $\mathcal{W}$ are independent we have:

$$ D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] \\ = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \left(\log \frac{\mathcal{N}_{0}(\pmb{\mu})}{\mathcal{N}_{1}(\pmb{\mu})} + \log \frac{\mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{W}_{1}(\pmb{\Lambda})}\right) d\pmb{\mu} d\pmb{\Lambda} $$

how can i proceed from here?


1 Answer 1


You are getting closed. To ease the derivation, let's redefine some notations:

\begin{cases} p(\pmb{\mu}, \pmb{\Lambda}) = p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{p}, (\lambda_{p} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{p}, \nu_{p} \right)\\ q(\pmb{\mu}, \pmb{\Lambda}) = q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{q}, (\lambda_{q} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{q}, \nu_{q} \right) \end{cases}

The KL divergence of interest is:

\begin{align} & D_{\mathrm{KL}} \left[ p(\pmb{\mu}, \pmb{\Lambda}) \Vert q(\pmb{\mu}, \pmb{\Lambda}) \right] \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu}, \pmb{\Lambda}) \ln \frac{p(\pmb{\mu}, \pmb{\Lambda})}{q(\pmb{\mu}, \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} + \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda}\\ & = \int_{\Lambda} p(\pmb{\Lambda}) \left[\int_{\mu} p(\pmb{\mu} \vert \pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} \right] d\pmb{\Lambda} + \int_{\Lambda} p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\Lambda}\\ & = \mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] + D_{\mathrm{KL}} \left[ p(\pmb{\Lambda}) \Vert q(\pmb{\Lambda}) \right].\\ & \tag{eq:KL_normal_wishart} \label{eq:KL_normal_wishart} \end{align}

To what I am aware of, there is a closed-form for the KL divergence between two Wishart distributions, corresponding to the second term. However, the first term is complicated, and I believe that further assumptions (eg. diagonal normal distributions) should be made to have a closed-form solution.

The first term is an expectation of the KL divergence between two normal distributions w.r.t. $p(\pmb{\Lambda})$, and also has a closed-form solution. To be specific, the KL divergence between 2 normal distributions can be written as:

\begin{aligned}[b] & D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \\ & = \frac{1}{2} \left[ \mathrm{tr}\left( \lambda_{q} \pmb{\Lambda} \lambda_{p}^{-1} \pmb{\Lambda}^{-1} \right) + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + \ln \frac{\mathrm{det}(\lambda_{p} \pmb{\Lambda})}{\mathrm{det}(\lambda_{q} \pmb{\Lambda})}\right] \\ & = \frac{1}{2} \left[ D \frac{\lambda_{q}}{\lambda_{p}} + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + D \ln \frac{\lambda_{p}}{\lambda_{q}} \right]. \end{aligned}

Note that for a Wishart distribution: $\mathbb{E}_{p(\pmb{\Lambda})} \left[ \pmb{\Lambda} \right] = \nu_{p} \mathbf{W}_{p}$. Hence, the first term of $\eqref{eq:KL_normal_wishart}$ can be obtained as:

\begin{aligned}[b] &\mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] \\ & = \frac{\lambda_{q}}{2} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \nu_{p} \mathbf{W}_{p} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) + \frac{D}{2} \left( \frac{\lambda_{q}}{\lambda_{p}} - \ln \frac{\lambda_{q}}{\lambda_{p}} - 1 \right). \end{aligned}

  • $\begingroup$ What exactly is a diagonal normal distribution? $\endgroup$ Dec 17, 2019 at 3:46
  • 1
    $\begingroup$ It is a multivariate normal distribution with diagonal covariance matrix. $\endgroup$
    – Cuong
    Dec 17, 2019 at 3:50
  • $\begingroup$ thanks! i will take a closer look and get back to you. $\endgroup$
    – hh32
    Dec 17, 2019 at 17:43
  • $\begingroup$ I have a question regarding the DKL part of the two normal distributions. How do you get from $\ln \frac{det(\lambda_p\Lambda)}{det(\lambda_q\Lambda)}$ to $D\ln\frac{\lambda_p}{\lambda_q}$? Also, shouldn't it be $\ln \frac{det(\lambda_q\Lambda)}{det(\lambda_p\Lambda)}$? $\endgroup$
    – hh32
    Dec 20, 2019 at 9:58
  • 1
    $\begingroup$ The covariance matrix in this case is: $(\lambda \pmb{\Lambda})^{-1}$. Hence, following the closed-form for KL divergence between 2 normal distributions, it should be something with p/q due to the inverse. Then, applying the property of matrix inverse: $\mathrm{det}(\alpha \mathbf{A}) = \alpha^{d} \mathrm{det}(\mathbf{A})$, where $d$ is the dimension of matrix $\mathbf{A}$. $\endgroup$
    – Cuong
    Dec 20, 2019 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.