# How to use Kullback-leibler divergence if mean and standard deviation of of two Gaussian Distribution is provided?

With Apache Spark MLLib library I am trying to find Clusters within a dataset by using Gaussian Mixture Model (number cluster =3) . Now it returns 3 different values of mean and standard deviation. I am trying to find that if there exists any overlappping between any two distribution. To do that, I am trying find the distances between the distribution.

Standard code for KL Div looks like this and generally takes the argument, two arrays of probabilities corresponding to two different distributions.

Now my question is 1. How do I change the equation to work on mean and sigma? 2. How do I come to the conclusion that the distributions are overlapping by looking at the return value?

• How many dimensions are you working in? If it is anything more than 1 then you should have CO-variance matrices. This allows the Gaussian to be elliptical and rotated with respect to the axes. There should always be overlap - the probability never goes to actual zero. It gets approaches zero in the limit of infinite distance. – EngrStudent Sep 13 '16 at 14:08
• Dataset consists of 1D data (only decimal numbers to be precise). – Avik Dutta Sep 13 '16 at 15:34
• The domain of the normal distribution is the real line. It goes from -infinity to +infinity. Every value on that line has a non-zero probability. What I think you are asking is the locations where the probability of being in one cluster is exactly equal to the probability of being in another cluster. Everything to one side of that line is going to be more likely member of one cluster. Everything on the other side is going to be more likely to belong to another cluster. – EngrStudent Sep 13 '16 at 18:52

You can compute pairwise KL divergence as a function of parameters in closed form for two Gaussian distributions $p$ and $q$. The uni-variate case:

$KL(p||q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_{1}^{2} + (\mu_1-\mu_2)^2}{2\sigma_{2}^{2}} - \frac{1}{2}$

and the multi-variate case:

$KL(p||q) = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - d + \text{tr} (\Sigma_2^{-1}\Sigma_1) + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]$

as derived here and here. Alternatively, you can try visualizing the cluster overlap by plotting the density of the mixture components.

To complete the answer given by Vadim, there are also many approximation of the Kullback-Leibler divergence between mixtures of Gaussian distributions.

These approximations are surprisingly easy to compute and implement. These paper by Hershey & Olsen proposes 7 or 8 different approximations and advise the use of the variational approximation: https://pdfs.semanticscholar.org/4f8d/eabc58014eae708c3e6ee27114535325067b.pdf (Paper title is: Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models.)

It will give you a similarity measure of the global mixture and you will not have to compare component by component.