0
$\begingroup$

I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding marginals can be expressed as (notation is from the book) I(1:2)=-0.5log(1-rho²) where rho is the correlation coefficient.

This particular KL divergence is the same as the mutual information and the result can be easily derived from the mutual information definition as for example can be seen in the linear correlation section of the wikipedia page: https://en.wikipedia.org/wiki/Mutual_information.

My problem comes with the derivation of the symmetric KL divergence (Jeffreys divergence) for the same distributions, on the same page (4.4) the book states that J(1,2)=rho²/(1-rho²) but what I get is simply 2xI(1:2), i.e. -log(1-rho²). So I am not able to derive the given result or see where it comes from. Please can you provide some help?

$\endgroup$
1

1 Answer 1

1
$\begingroup$

Ok, I have solved a simplified version using (1.3) from page 190 of Kullback's book (Dover 1997 ed.) and assuming equal population means and variances and noting that the product of marginals has zero covariance. Sorry for not putting the equations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.