1
$\begingroup$

What is the correct mathematical expression for computing Jensen-Shannon divergence between multiple probability distributions?

I found the following expression on Wikipedia, but I did not find any official reference:

$$JS-Divergence(p^1,...,p^m)=H(\frac{p^1+...+p^m}{m})-\frac{\sum_{j=1}^{m} H(p^j)}{m}$$

where $H$ is the Shannon-entropy.

What is the intuition of divergence in multiple distribution case? In case of two distribution P & Q, we can interpret is as a distance between them JS(P||Q). So in multi-distribution case, it is the distance between which distributions?

Is there an efficient Python code available for it?

$\endgroup$
3
$\begingroup$

I found one paper that uses the JS-Divergence of multiple distributions to estimate the hardness of a query (in the area of Information Retrieval). The paper can be found here and they themselves refer a paper called "Divergence measures based on the shannon entropy (1991)". They also give a little bit different mathematical expression for it.

As for the interpretation, they explain it as follows:

Given a set of distributions thus obtained, we employ the well known Jensen-Shannon divergence [8] to measure the diversity of the distributions corresponding [...] So the Jensen-Shannon divergence can be seen to measure the overall diversity between all the probability distributions.

As for the Python code, I couldn't find any package that implements the JSD for more than two distributions. But there is already one quite straightforward code example on crossvalidated (see here)

.

$\endgroup$
  • $\begingroup$ @ Bobipuegi Python code you referred is for 2 distributions only while I was looking for a general case. Anyway, I can write it myself as I have done for 2 distributions. Also, the paper you referred does not give exactly the same expression as given in my question, so I am not sure if it is correct (though I have seen on Stack Exchange someone using above expression). I want to make sure which expression is the correct one. More importantly, my question is about understanding intuition behind multi-distribution JS divergence. Between which distributions does it compute distance? $\endgroup$ – Hello World Sep 20 '17 at 17:15
  • $\begingroup$ 1. I cannot do the math right now, but it seems to me that the formula you give and the one in the paper are equivalent after a bit of reshaping. 2. the Python code in the link I attached is able to handle multiple distributions 3. as cited in the paper, the js-divergence of multiple distributions isn't a distance measure but an overall diversity measure $\endgroup$ – Bobipuegi Sep 20 '17 at 17:41
  • $\begingroup$ @ Bobipuegi So is it JS(P1 vs. P2, P3,...,Pn) or JS(P1 vs. P2 vs. P3 vs....vs., Pn) or something like this? $\endgroup$ – Hello World Sep 20 '17 at 18:00
  • $\begingroup$ @ Bobipuegi For 2 distributions, it is JS(P vs. Q) $\endgroup$ – Hello World Sep 20 '17 at 18:07
  • $\begingroup$ I understand it as first calculating a "centroid" distribution and then calculating something like JS(P1vs. Centroid, P2 vs. Centroid .... Pn vs. Centroid) $\endgroup$ – Bobipuegi Sep 20 '17 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.