# How to write down Information Gain

So I have learnt about Entropy and how to do it on paper, but I get confused when I try to apply Information Gain.

Say I have a set of letter freqency classifications:

a = 5
b = 5
c = 10


I can perform entropy on these numbers fine, but the formula for Information Gain confuses me and I can find little info on how to use it, unlike Entropy.

I am having trouble finding the same forumula as in the book im reading 'Machine Learning' by Tom Mitchell, but I cant seem to follow the example.

How would I apply Information Gain on the classifications above?

• Could you post the formula from the book here? That will make your question self-contained and allows people when don't have the book (available to them) to answer the question. Nov 1, 2016 at 9:47

$$I(X;Y)=\sum _{y\in Y}\sum _{x\in X}p(x,y)\log {\left({\frac {p(x,y)}{p(x)\,p(y)}}\right)}\!$$
We can however, calculate the entropy $$H = \frac{5}{20}\log{\frac{5}{20}}+\frac{5}{20}\log{\frac{5}{20}}+\frac{10}{20}\log{\frac{10}{20}}$$