For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is infinity. I have read that KL divergence is always finite if Q is continuous in respect to P, but I am having a hard time understanding why, and how to know what the maximum KL divergence between the product of the marginals and the joint is for any given marginal distributions. I have a feeling that the maximum possible mutual information between two distributions is related to the KL divergence of their marginal distributions - i.e. a uniformly distributed variable cannot have infinite mutual information with a Gaussian distributed variable, and that is because the KL divergence between two two marginals is not 0. However, if I know the KL divergence of two marginal distributions, is it possible to calculate the maximum possible mutual information between them?

I hope this isn't a stupid question, but I know there must be a relationship and I can't find anything about it readily.


  • $\begingroup$ It might help to distinguish "infinite" from undefined from unbounded. For example, in the case of a Uniform$[0,1]$ distribution relative to Gaussian distributions, (1) the KL divergence from the Gaussian to the Uniform is undefined (due to the zero values of the Uniform PDF at points where all Gaussians have positive density) and (2) although the KL divergence from the Uniform to any Gaussian is finite, there is no upper bound to the divergence among all Gaussian distributions. $\endgroup$ – whuber Jun 30 '14 at 19:34
  • $\begingroup$ When two random variables linear related, the mutual information go infinity $\endgroup$ – Bùi Nhật Duy Sep 3 '18 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.