Information gain, mutual information and related measures

Andrew More defines information gain as:

$IG(Y|X) = H(Y) - H(Y|X)$

where $H(Y|X)$ is the conditional entropy. However, Wikipedia calls the above quantity mutual information.

Wikipedia on the other hand defines information gain as the Kullback–Leibler divergence (aka information divergence or relative entropy) between two random variables:

$D_{KL}(P||Q) = H(P,Q) - H(P)$

where $H(P,Q)$ is defined as the cross-entropy.

These two definitions seem to be inconsistent with each other.

I have also seen other authors talking about two additional related concepts, namely differential entropy and relative information gain.

What is the precise definition or relationship between these quantities? Is there a good text book that covers them all?

• Information gain
• Mutual information
• Cross entropy
• Conditional entropy
• Differential entropy
• Relative information gain
• To further add to confusion, note that the notation you used for cross entropy is also the same notation used for the joint entropy. I've used $H^x(P, Q)$ for the cross-entropy to avoid confusing myself, but that's for my benefit and I've never seen that notation elsewhere. – Michael McGowan Jul 25 '11 at 14:25

I think that calling the Kullback-Leibler divergence "information gain" is non-standard.

The first definition is standard.

EDIT: However, $H(Y)−H(Y|X)$ can also be called mutual information.

Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea.

Textbooks: see "Good introduction into different kinds of entropy".

Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015

Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html

David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html

• Thanks @wolf. I'm inclined to accept this answer. If the first definition is standard, how would you define mutual information? – Amelio Vazquez-Reina Jul 23 '11 at 20:42
• sorry. the first quantity, $IG(Y|X)=H(Y)−H(Y|X)$ is also often called mutual information. That's a case of inconsistent naming. As I said, I don't think there is any consistent, unambiguous, one-to-one correspondence of concepts and names. E.g. "mutual information" or "info gain" is a special case of K-L divergence, so that that wikipedia article is not that far off. – wolf.rauch Jul 25 '11 at 4:06

Mutual information can be defined using Kullback-Liebler as \begin{align*} I(X;Y) = D_{KL}(p(x,y)||p(x)p(y)). \end{align*}