So I have learnt about Entropy and how to do it on paper, but I get confused when I try to apply Information Gain.

Say I have a set of letter freqency classifications:

a = 5
b = 5
c = 10

I can perform entropy on these numbers fine, but the formula for Information Gain confuses me and I can find little info on how to use it, unlike Entropy.

I am having trouble finding the same forumula as in the book im reading 'Machine Learning' by Tom Mitchell, but I cant seem to follow the example.

How would I apply Information Gain on the classifications above?

  • $\begingroup$ Could you post the formula from the book here? That will make your question self-contained and allows people when don't have the book (available to them) to answer the question. $\endgroup$
    – dimpol
    Nov 1, 2016 at 9:47

1 Answer 1


you can think of Information gain as a "covariance" for categorical data, the more correlated the two variables, the larger the mutual informaion.

$${\displaystyle I(X;Y)=\sum _{y\in Y}\sum _{x\in X}p(x,y)\log {\left({\frac {p(x,y)}{p(x)\,p(y)}}\right)}\!}$$

As you can see, we are missing from your data the reference point.

Information gain measures how many bits of information the variable (in your case class frequencies) adds to a predefined distribution.

We are missing that predefined distribution

We can however, calculate the entropy $$H = \frac{5}{20}\log{\frac{5}{20}}+\frac{5}{20}\log{\frac{5}{20}}+\frac{10}{20}\log{\frac{10}{20}}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.