80
$\begingroup$

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math stats, so I'm having some trouble extending it to the multivariate case. I'm sure I'm just missing something simple.

Here's what I have...

Suppose both $p$ and $q$ are the pdfs of normal distributions with means $\mu_1$ and $\mu_2$ and variances $\Sigma_1$ and $\Sigma_2$, respectively. The Kullback-Leibler distance from $q$ to $p$ is:

$\int \left[\log( p(x)) - \log( q(x)) \right]\ p(x)\ dx$, which for two multivariate normals is:

$\frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - d + Tr(\Sigma_2^{-1}\Sigma_1) + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]$

Following the same logic as this proof, I get to about here before I get stuck:

$=\int \left[ \frac{d}{2} \log\frac{|\Sigma_2|}{|\Sigma_1|} + \frac{1}{2} \left((x-\mu_2)^T\Sigma_2^{-1}(x-\mu_2) - (x-\mu_1)^T\Sigma_2^{-1}(x-\mu_1) \right) \right] \times p(x) dx$

$=\mathbb{E} \left[ \frac{d}{2} \log\frac{|\Sigma_2|}{|\Sigma_1|} + \frac{1}{2} \left((x-\mu_2)^T\Sigma_2^{-1}(x-\mu_2) - (x-\mu_1)^T\Sigma_2^{-1}(x-\mu_1) \right) \right]$

I think I have to implement the trace trick, but I'm just not sure what to do after that. Any helpful hints to put me back on the right track would be appreciated!

$\endgroup$
1

1 Answer 1

87
$\begingroup$

Starting with where you began with some slight corrections, we can write

$$ \begin{aligned} KL &= \int \left[ \frac{1}{2} \log\frac{|\Sigma_2|}{|\Sigma_1|} - \frac{1}{2} (x-\mu_1)^T\Sigma_1^{-1}(x-\mu_1) + \frac{1}{2} (x-\mu_2)^T\Sigma_2^{-1}(x-\mu_2) \right] \times p(x) dx \\ &= \frac{1}{2} \log\frac{|\Sigma_2|}{|\Sigma_1|} - \frac{1}{2} \text{tr}\ \left\{E[(x - \mu_1)(x - \mu_1)^T] \ \Sigma_1^{-1} \right\} + \frac{1}{2} E[(x - \mu_2)^T \Sigma_2^{-1} (x - \mu_2)] \\ &= \frac{1}{2} \log\frac{|\Sigma_2|}{|\Sigma_1|} - \frac{1}{2} \text{tr}\ \{I_d \} + \frac{1}{2} (\mu_1 - \mu_2)^T \Sigma_2^{-1} (\mu_1 - \mu_2) + \frac{1}{2} \text{tr} \{ \Sigma_2^{-1} \Sigma_1 \} \\ &= \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - d + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]. \end{aligned} $$

Note that I have used a couple of properties from Section 8.2 of the Matrix Cookbook.

$\endgroup$
7
  • $\begingroup$ I see you took out the D that I had originally. Wouldn't you have a D term after taking the log of the Gaussian in the first few steps? $\endgroup$
    – dmartin
    Commented Jun 3, 2013 at 15:19
  • 1
    $\begingroup$ Consider the scaling factor $(2\pi)^{-d/2} |\Sigma_k|^{-1/2}$, $k = 1,2$ of the multivariate normal density. When computing the log-difference, the $(2\pi)^{-d/2}$ term goes away. There is no $d$ term for the determinants -- simply, a $1/2$, which is factored out. $\endgroup$
    – ramhiser
    Commented Jun 3, 2013 at 15:33
  • $\begingroup$ Hi, how did you come up with the last step? How did you change sign of $\mu_1 - \mu_2$ into $\mu_2 - \mu_1$? $\endgroup$
    – acidghost
    Commented Apr 11, 2016 at 6:10
  • 1
    $\begingroup$ @acidghost Either one works because we can factor out a negative one from both sides. Multiplying the two negative ones yields a positive one. $\endgroup$
    – ramhiser
    Commented Apr 12, 2016 at 0:06
  • $\begingroup$ @JohnA.Ramey. Which property of the sec 8.2 you used? Is it eq. 380 ? If yes, I'm not able to follow, Can you explain? $\endgroup$
    – CKM
    Commented Aug 23, 2017 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.