I'm trying to calculate the Jensen-Shannon divergence between two multivariate Gaussians.
I found a closed-form expression both for the KL divergence and JS divergence between two Gaussians in this paper: Here
(Sorry I'm new to stackexchange so I don't know how to type in math equations)
For KL divergence between two Gaussians: See equation (76) on page 12
For JS divergence between two Gaussians: See equation (83) on page 13
I implemented these two equations in matlab as follows:
function gJS = gaussian_gJS(d, mu1, sigma1, mu2, sigma2) alpha = 0.5; sigma1_inv = inv(sigma1); sigma2_inv = inv(sigma2); M1_inv = inv(-0.5*sigma1_inv); M2_inv = inv(-0.5*sigma2_inv); Sigma_a = inv( (1 - alpha)*sigma1_inv + alpha*sigma2_inv ); Mu_a = Sigma_a * ( (1 - alpha)*M1_inv*mu1 + alpha*M2_inv*mu2 ); gJS = (1 - alpha)*gaussian_KL(d, mu1, sigma1, Mu_a, Sigma_a) + ... alpha*gaussian_KL(d, mu2, sigma2, Mu_a, Sigma_a); end function KL_Divergence = gaussian_KL(d, mu1, sigma1, mu2, sigma2) inv_sigma2 = inv(sigma2); KL_Divergence = 0.5*trace(inv_sigma2*sigma1) + (mu2 - mu1)'*inv_sigma2 + log(det(sigma2)/det(sigma1)) - d; end
In the paper, the author gives a numerical example (on page 13, below equation 84) where he calculates the gJS for some example:
gJS( mu_1 = -1, sigma_1 = 4, mu_2 = -4, sigma_2 = 16) = 0.4087
However when I calculate it using my implementation, I end up with a value of around 67.94.
One good thing is that my function is symmetric around inputs so at least that part is working. I've double checked my implementation, and I don't see any differences between my code and his equations. Further, if I calculate the KL-divergence using my function, for a Gaussian against itself, I don't get zero, instead I get -0.5. I've seen another thread on stackexchange where they derived the KL divergence between two multivariate Gaussians and their expression was slightly different than Eqn. (76) in the paper I linked.
Anyone have any thoughts?