5
$\begingroup$

With Apache Spark MLLib library I am trying to find Clusters within a dataset by using Gaussian Mixture Model (number cluster =3) . Now it returns 3 different values of mean and standard deviation. I am trying to find that if there exists any overlappping between any two distribution. To do that, I am trying find the distances between the distribution.

Standard code for KL Div looks like this and generally takes the argument, two arrays of probabilities corresponding to two different distributions.

Now my question is 1. How do I change the equation to work on mean and sigma? 2. How do I come to the conclusion that the distributions are overlapping by looking at the return value?

Improve this question
$\endgroup$
3
  • $\begingroup$ How many dimensions are you working in? If it is anything more than 1 then you should have CO-variance matrices. This allows the Gaussian to be elliptical and rotated with respect to the axes. There should always be overlap - the probability never goes to actual zero. It gets approaches zero in the limit of infinite distance. $\endgroup$
    – EngrStudent
    Sep 13, 2016 at 14:08
  • $\begingroup$ Dataset consists of 1D data (only decimal numbers to be precise). $\endgroup$
    – Avik Dutta
    Sep 13, 2016 at 15:34
  • 1
    $\begingroup$ The domain of the normal distribution is the real line. It goes from -infinity to +infinity. Every value on that line has a non-zero probability. What I think you are asking is the locations where the probability of being in one cluster is exactly equal to the probability of being in another cluster. Everything to one side of that line is going to be more likely member of one cluster. Everything on the other side is going to be more likely to belong to another cluster. $\endgroup$
    – EngrStudent
    Sep 13, 2016 at 18:52

2 Answers 2

Reset to default
7
$\begingroup$

You can compute pairwise KL divergence as a function of parameters in closed form for two Gaussian distributions $p$ and $q$. The uni-variate case:

$KL(p||q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_{1}^{2} + (\mu_1-\mu_2)^2}{2\sigma_{2}^{2}} - \frac{1}{2}$

and the multi-variate case:

$KL(p||q) = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - d + \text{tr} (\Sigma_2^{-1}\Sigma_1) + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]$

as derived here and here. Alternatively, you can try visualizing the cluster overlap by plotting the density of the mixture components.

Improve this answer
$\endgroup$
1
$\begingroup$

To complete the answer given by Vadim, there are also many approximation of the Kullback-Leibler divergence between mixtures of Gaussian distributions.

These approximations are surprisingly easy to compute and implement. These paper by Hershey & Olsen proposes 7 or 8 different approximations and advise the use of the variational approximation: https://pdfs.semanticscholar.org/4f8d/eabc58014eae708c3e6ee27114535325067b.pdf (Paper title is: Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models.)

It will give you a similarity measure of the global mixture and you will not have to compare component by component.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.