How to write down Information Gain

Question

So I have learnt about Entropy and how to do it on paper, but I get confused when I try to apply Information Gain.

Say I have a set of letter freqency classifications:

a = 5
b = 5
c = 10

I can perform entropy on these numbers fine, but the formula for Information Gain confuses me and I can find little info on how to use it, unlike Entropy.

I am having trouble finding the same forumula as in the book im reading 'Machine Learning' by Tom Mitchell, but I cant seem to follow the example.

How would I apply Information Gain on the classifications above?

Answer 1

you can think of Information gain as a "covariance" for categorical data, the more correlated the two variables, the larger the mutual informaion.

$${\displaystyle I(X;Y)=\sum _{y\in Y}\sum _{x\in X}p(x,y)\log {\left({\frac {p(x,y)}{p(x)\,p(y)}}\right)}\!}$$

As you can see, we are missing from your data the reference point.

Information gain measures how many bits of information the variable (in your case class frequencies) adds to a predefined distribution.

We are missing that predefined distribution

We can however, calculate the entropy $$H = \frac{5}{20}\log{\frac{5}{20}}+\frac{5}{20}\log{\frac{5}{20}}+\frac{10}{20}\log{\frac{10}{20}}$$