Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

 4 replaced http://stackoverflow.com/ with https://stackoverflow.com/ edited May 23 '17 at 12:39 I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html 3 replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/ edited Apr 13 '17 at 12:44 I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html 2 content edit (see comments) edited Jul 25 '11 at 4:08 wolf.rauch 1,8371414 silver badges1111 bronze badges I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html I think that calling the Kullback-Leibler divergence "information gain" is non-standard. The first definition is standard. EDIT: However, $$H(Y)−H(Y|X)$$ can also be called mutual information. Note that I don't think you will find any scientific discipline that really has a standardized, precise, and consistent naming scheme. So you will always have to look at the formulae, because they will generally give you a better idea. Textbooks: see "Good introduction into different kinds of entropy". Also: Cosma Shalizi: Methods and Techniques of Complex Systems Science: An Overview, chapter 1 (pp. 33--114) in Thomas S. Deisboeck and J. Yasha Kresh (eds.), Complex Systems Science in Biomedicine http://arxiv.org/abs/nlin.AO/0307015 Robert M. Gray: Entropy and Information Theory http://ee.stanford.edu/~gray/it.html David MacKay: Information Theory, Inference, and Learning Algorithms http://www.inference.phy.cam.ac.uk/mackay/itila/book.html 1 answered Jul 22 '11 at 21:11 wolf.rauch 1,8371414 silver badges1111 bronze badges