# Jensen-Shannon divergence calculation for 3 prob distributions: Is this ok?

I would like to calculate the jensen-shannon divergence for he following 3 distributions. Is the calculation below correct? (I followed the JSD formula from wikipedia):

P1  a:1/2  b:1/2    c:0
P2  a:0    b:1/10   c:9/10
P3  a:1/3  b:1/3    c:1/3
All distributions have equal weights, ie 1/3.

JSD(P1, P2, P3) = H[(1/6, 1/6, 0) + (0, 1/30, 9/30) + (1/9,1/9,1/9)] -
[1/3*H[(1/2,1/2,0)] + 1/3*H[(0,1/10,9/10)] + 1/3*H[(1/3,1/3,1/3)]]

JSD(P1, P2, P3) = H[(1/6, 1/5, 9/30)] - [0 + 1/3*0.693 + 0] = 1.098-0.693 = 0.867


EDIT Here's some simple dirty Python code that calculates this as well:

    def entropy(prob_dist, base=math.e):
return -sum([p * math.log(p,base) for p in prob_dist if p != 0])

def jsd(prob_dists, base=math.e):
weight = 1/len(prob_dists) #all same weight
js_left = [0,0,0]
js_right = 0
for pd in prob_dists:
js_left[0] += pd[0]*weight
js_left[1] += pd[1]*weight
js_left[2] += pd[2]*weight
js_right += weight*entropy(pd,base)
return entropy(js_left)-js_right

usage: jsd([[1/2,1/2,0],[0,1/10,9/10],[1/3,1/3,1/3]])

-
Nice Python code by the way! – gui11aume Jun 1 '12 at 23:55

There is mistake in the mixture distribution. It should be $(5/18, 28/90, 37/90)$ instead of $(1/6, 1/5, 9/30)$ which does not sum up to 1. The entropy (with natural log) of that is 1.084503. Your other entropy terms are wrong.

I will give the detail of one computation:

$$H(1/2,1/2,0) = -1/2*\log(1/2) - 1/2*\log(1/2) + 0 = 0.6931472$$

In a similar way, the other terms are 0.325083 and 1.098612. So the final result is 1.084503 - (0.6931472 + 0.325083 + 1.098612)/3 = 0.378889

-
+1. Quick and dirty R calculation: h <- function(x) {h <- function(x) {y <- x[x > 0]; -sum(y * log(y))}; jsd <- function(p,q) {h(q %*% p) - q %*% apply(p, 2, h)}. Argument p is a matrix whose rows are the distributions and argument q is the vector of weights. E.g., p <- matrix(c(1/2,1/2,0, 0,1/10,9/10, 1/3,1/3,1/3), ncol=3, byrow=TRUE); q <- c(1/3,1/3,1/3); jsd(p,q) returns $0.378889$ (which approximates the log of $3^{34/15} 5^{1/9} 2^{-13/45} 7^{-14/45} 37^{-37/90}$). – whuber May 31 '12 at 19:05
Not so dirty... ;-) – gui11aume May 31 '12 at 22:19
Thanks @whuber and @gui11aume! Just last questions: (1) The second term of the mixture shouldn't be 1/6+1/30+1/9 = 23/90 instead of 28/90 ? (2) The log must be the natural one ? and (3) The meaning of the result of of the JSD is supposed to be the average distance between all distributions ? – kanzen_master May 31 '12 at 23:23
(1) Redo the math. (2) Entropy can be measured using any base of logarithm you like, as long as you are consistent. Natural, common, and base-2 logs are all conventional. (3) It's really a mean discrepancy between the distributions and their average. If you think of each distribution as a point, they form a cloud. You're looking at the average "distance" between the center of the cloud and its points, kind of like a mean radius. Intuitively, it measures the cloud's size. – whuber May 31 '12 at 23:51
@Legend I think you're right. I did not test sufficiently after finding that one result agreed with the answer I obtained in another way (with Mathematica). – whuber Jan 17 at 3:26