I'm having trouble to compute the KL Divergence between two normal-Wishart distributions. KL divergence from $Q$ to $P$ is defined as:

$$D_{\mathrm{KL}}(P \Vert Q) = \int p(x) \log \frac{p(x)}{q(x)}dx$$

and a normal-Wishart distribution is defined as:

$$f(\pmb{\mu},\pmb{\Lambda} \vert \pmb{\mu}_0, \lambda, \mathbf{W}, \nu) = \mathcal{N} \left(\pmb{\mu} \vert \pmb{\mu}_0, (\lambda \pmb{\Lambda})^{-1} \right) \mathcal{W}(\pmb{\Lambda} \vert \mathbf{W}, \nu) $$

where $\pmb{\mu}_0 \in \mathbb{R}^D, \mathbf{W} \in \mathbb{R}^{D\times D},\nu > D - 1,\lambda > 0$ are the parameters of the distribution. So we have:

$$D_{\mathrm{KL}} (f_0 \Vert f_1) = \int f_0(x) \log \frac{f_0(x)}{f_1(x)}dx$$

$$D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \log \frac{\mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda})}d\pmb{\mu}d\pmb{\Lambda}$$

because $\mathcal{N}$ and $\mathcal{W}$ are independent we have:

$$ D_{\mathrm{KL}} \left[ \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \Vert \mathcal{N}_{1}(\pmb{\mu}) \mathcal{W}_{1}(\pmb{\Lambda}) \right] \\ = \int \mathcal{N}_{0}(\pmb{\mu}) \mathcal{W}_{0}(\pmb{\Lambda}) \left(\log \frac{\mathcal{N}_{0}(\pmb{\mu})}{\mathcal{N}_{1}(\pmb{\mu})} + \log \frac{\mathcal{W}_{0}(\pmb{\Lambda})}{\mathcal{W}_{1}(\pmb{\Lambda})}\right) d\pmb{\mu} d\pmb{\Lambda} $$

how can i proceed from here?


You are getting closed. To ease the derivation, let's redefine some notations:

\begin{cases} p(\pmb{\mu}, \pmb{\Lambda}) = p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{p}, (\lambda_{p} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{p}, \nu_{p} \right)\\ q(\pmb{\mu}, \pmb{\Lambda}) = q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda}) & = \mathcal{N}\left( \pmb{\mu} \vert \pmb{\mu}_{q}, (\lambda_{q} \pmb{\Lambda})^{-1} \right) \mathcal{W} \left( \pmb{\Lambda} \vert \mathbf{W}_{q}, \nu_{q} \right) \end{cases}

The KL divergence of interest is:

\begin{align} & D_{\mathrm{KL}} \left[ p(\pmb{\mu}, \pmb{\Lambda}) \Vert q(\pmb{\mu}, \pmb{\Lambda}) \right] \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu}, \pmb{\Lambda}) \ln \frac{p(\pmb{\mu}, \pmb{\Lambda})}{q(\pmb{\mu}, \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda}) q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} \\ & = \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda} + \int_{\mu} \int_{\Lambda} p(\pmb{\mu} \vert \pmb{\Lambda}) p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\mu} d\pmb{\Lambda}\\ & = \int_{\Lambda} p(\pmb{\Lambda}) \left[\int_{\mu} p(\pmb{\mu} \vert \pmb{\Lambda}) \ln \frac{p(\pmb{\mu} \vert \pmb{\Lambda})}{q(\pmb{\mu} \vert \pmb{\Lambda})} d\pmb{\mu} \right] d\pmb{\Lambda} + \int_{\Lambda} p(\pmb{\Lambda}) \ln \frac{p(\pmb{\Lambda})}{q(\pmb{\Lambda})} d\pmb{\Lambda}\\ & = \mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] + D_{\mathrm{KL}} \left[ p(\pmb{\Lambda}) \Vert q(\pmb{\Lambda}) \right].\\ & \tag{eq:KL_normal_wishart} \label{eq:KL_normal_wishart} \end{align}

To what I am aware of, there is a closed-form for the KL divergence between two Wishart distributions, corresponding to the second term. However, the first term is complicated, and I believe that further assumptions (eg. diagonal normal distributions) should be made to have a closed-form solution.

The first term is an expectation of the KL divergence between two normal distributions w.r.t. $p(\pmb{\Lambda})$, and also has a closed-form solution. To be specific, the KL divergence between 2 normal distributions can be written as:

\begin{aligned}[b] & D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \\ & = \frac{1}{2} \left[ \mathrm{tr}\left( \lambda_{q} \pmb{\Lambda} \lambda_{p}^{-1} \pmb{\Lambda}^{-1} \right) + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + \ln \frac{\mathrm{det}(\lambda_{p} \pmb{\Lambda})}{\mathrm{det}(\lambda_{q} \pmb{\Lambda})}\right] \\ & = \frac{1}{2} \left[ D \frac{\lambda_{q}}{\lambda_{p}} + \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \lambda_{q} \pmb{\Lambda} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) - D + D \ln \frac{\lambda_{p}}{\lambda_{q}} \right]. \end{aligned}

Note that for a Wishart distribution: $\mathbb{E}_{p(\pmb{\Lambda})} \left[ \pmb{\Lambda} \right] = \nu_{p} \mathbf{W}_{p}$. Hence, the first term of $\eqref{eq:KL_normal_wishart}$ can be obtained as:

\begin{aligned}[b] &\mathbb{E}_{p(\pmb{\Lambda})} \left[ D_{\mathrm{KL}} \left[ p(\pmb{\mu} \vert \pmb{\Lambda}) \Vert q(\pmb{\mu} \vert \pmb{\Lambda}) \right] \right] \\ & = \frac{\lambda_{q}}{2} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right)^{\top} \nu_{p} \mathbf{W}_{p} \left( \pmb{\mu}_{q} - \pmb{\mu}_{p} \right) + \frac{D}{2} \left( \frac{\lambda_{q}}{\lambda_{p}} - \ln \frac{\lambda_{q}}{\lambda_{p}} - 1 \right). \end{aligned}

| cite | improve this answer | |
  • $\begingroup$ What exactly is a diagonal normal distribution? $\endgroup$ – SOULed_Outt Dec 17 '19 at 3:46
  • 1
    $\begingroup$ It is a multivariate normal distribution with diagonal covariance matrix. $\endgroup$ – Cuong Dec 17 '19 at 3:50
  • $\begingroup$ thanks! i will take a closer look and get back to you. $\endgroup$ – hh32 Dec 17 '19 at 17:43
  • $\begingroup$ I have a question regarding the DKL part of the two normal distributions. How do you get from $\ln \frac{det(\lambda_p\Lambda)}{det(\lambda_q\Lambda)}$ to $D\ln\frac{\lambda_p}{\lambda_q}$? Also, shouldn't it be $\ln \frac{det(\lambda_q\Lambda)}{det(\lambda_p\Lambda)}$? $\endgroup$ – hh32 Dec 20 '19 at 9:58
  • 1
    $\begingroup$ The covariance matrix in this case is: $(\lambda \pmb{\Lambda})^{-1}$. Hence, following the closed-form for KL divergence between 2 normal distributions, it should be something with p/q due to the inverse. Then, applying the property of matrix inverse: $\mathrm{det}(\alpha \mathbf{A}) = \alpha^{d} \mathrm{det}(\mathbf{A})$, where $d$ is the dimension of matrix $\mathbf{A}$. $\endgroup$ – Cuong Dec 20 '19 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.