1
$\begingroup$

I'm trying to calculate the Jensen-Shannon divergence between two multivariate Gaussians.

I found a closed-form expression both for the KL divergence and JS divergence between two Gaussians in this paper: Here

(Sorry I'm new to stackexchange so I don't know how to type in math equations)

  • For KL divergence between two Gaussians: See equation (76) on page 12

  • For JS divergence between two Gaussians: See equation (83) on page 13

I implemented these two equations in matlab as follows:

function gJS = gaussian_gJS(d, mu1, sigma1, mu2, sigma2)

alpha = 0.5;

sigma1_inv = inv(sigma1);
sigma2_inv = inv(sigma2);

M1_inv         = inv(-0.5*sigma1_inv);
M2_inv         = inv(-0.5*sigma2_inv);

Sigma_a = inv( (1 - alpha)*sigma1_inv + alpha*sigma2_inv );
Mu_a    = Sigma_a * ( (1 - alpha)*M1_inv*mu1 + alpha*M2_inv*mu2 );

gJS = (1 - alpha)*gaussian_KL(d, mu1, sigma1, Mu_a, Sigma_a) + ...
            alpha*gaussian_KL(d, mu2, sigma2, Mu_a, Sigma_a);
end

function KL_Divergence = gaussian_KL(d, mu1, sigma1, mu2, sigma2)

inv_sigma2 = inv(sigma2);

KL_Divergence = 0.5*trace(inv_sigma2*sigma1) + (mu2 - mu1)'*inv_sigma2 + log(det(sigma2)/det(sigma1)) - d;

end

In the paper, the author gives a numerical example (on page 13, below equation 84) where he calculates the gJS for some example:

 gJS( mu_1 = -1, sigma_1 = 4, mu_2 = -4, sigma_2 = 16) = 0.4087

However when I calculate it using my implementation, I end up with a value of around 67.94.

One good thing is that my function is symmetric around inputs so at least that part is working. I've double checked my implementation, and I don't see any differences between my code and his equations. Further, if I calculate the KL-divergence using my function, for a Gaussian against itself, I don't get zero, instead I get -0.5. I've seen another thread on stackexchange where they derived the KL divergence between two multivariate Gaussians and their expression was slightly different than Eqn. (76) in the paper I linked.

Anyone have any thoughts?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.