Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
3

I would like to calculate the jensen-shannon divergence for he following 3 distributions. Is the calculation below correct? (I followed the JSD formula from wikipedia):

P1  a:1/2  b:1/2    c:0
P2  a:0    b:1/10   c:9/10
P3  a:1/3  b:1/3    c:1/3
All distributions have equal weights, ie 1/3.

JSD(P1, P2, P3) = H[(1/6, 1/6, 0) + (0, 1/30, 9/30) + (1/9,1/9,1/9)] - 
                 [1/3*H[(1/2,1/2,0)] + 1/3*H[(0,1/10,9/10)] + 1/3*H[(1/3,1/3,1/3)]]

JSD(P1, P2, P3) = H[(1/6, 1/5, 9/30)] - [0 + 1/3*0.693 + 0] = 1.098-0.693 = 0.867

Thanks in advance...

EDIT Here's some simple dirty Python code that calculates this as well:

    def entropy(prob_dist, base=math.e):
        return -sum([p * math.log(p,base) for p in prob_dist if p != 0])

    def jsd(prob_dists, base=math.e):
        weight = 1/len(prob_dists) #all same weight
        js_left = [0,0,0]
        js_right = 0    
        for pd in prob_dists:
            js_left[0] += pd[0]*weight
            js_left[1] += pd[1]*weight
            js_left[2] += pd[2]*weight
            js_right += weight*entropy(pd,base)
        return entropy(js_left)-js_right

usage: jsd([[1/2,1/2,0],[0,1/10,9/10],[1/3,1/3,1/3]])
share|improve this question
1  
Nice Python code by the way! – gui11aume Jun 1 '12 at 23:55

1 Answer

up vote 9 down vote accepted

There is mistake in the mixture distribution. It should be $(5/18, 28/90, 37/90)$ instead of $(1/6, 1/5, 9/30)$ which does not sum up to 1. The entropy (with natural log) of that is 1.084503. Your other entropy terms are wrong.

I will give the detail of one computation:

$$H(1/2,1/2,0) = -1/2*\log(1/2) - 1/2*\log(1/2) + 0 = 0.6931472$$

In a similar way, the other terms are 0.325083 and 1.098612. So the final result is 1.084503 - (0.6931472 + 0.325083 + 1.098612)/3 = 0.378889

share|improve this answer
2  
+1. Quick and dirty R calculation: h <- function(x) {h <- function(x) {y <- x[x > 0]; -sum(y * log(y))}; jsd <- function(p,q) {h(q %*% p) - q %*% apply(p, 2, h)}. Argument p is a matrix whose rows are the distributions and argument q is the vector of weights. E.g., p <- matrix(c(1/2,1/2,0, 0,1/10,9/10, 1/3,1/3,1/3), ncol=3, byrow=TRUE); q <- c(1/3,1/3,1/3); jsd(p,q) returns $0.378889$ (which approximates the log of $3^{34/15} 5^{1/9} 2^{-13/45} 7^{-14/45} 37^{-37/90}$). – whuber♦ May 31 '12 at 19:05
1  
Not so dirty... ;-) – gui11aume May 31 '12 at 22:19
Thanks @whuber and @gui11aume! Just last questions: (1) The second term of the mixture shouldn't be 1/6+1/30+1/9 = 23/90 instead of 28/90 ? (2) The log must be the natural one ? and (3) The meaning of the result of of the JSD is supposed to be the average distance between all distributions ? – kanzen_master May 31 '12 at 23:23
3  
(1) Redo the math. (2) Entropy can be measured using any base of logarithm you like, as long as you are consistent. Natural, common, and base-2 logs are all conventional. (3) It's really a mean discrepancy between the distributions and their average. If you think of each distribution as a point, they form a cloud. You're looking at the average "distance" between the center of the cloud and its points, kind of like a mean radius. Intuitively, it measures the cloud's size. – whuber♦ May 31 '12 at 23:51
1  
@Legend I think you're right. I did not test sufficiently after finding that one result agreed with the answer I obtained in another way (with Mathematica). – whuber♦ Jan 17 at 3:26
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.