Setting: Multi-class classification problem with discrete nominal features.
There are many references mentioning the use of
IG(Information Gain) and
MI (Mutual Information) as measure of feature relevancy for filter-based feature selection. However, from the information-theoretic viewpoint it's not completely clear to me what is the difference between these two (and if there is any):
IG(X,Y) ?=? MI(X,Y) = H(X)-H(X|Y) = H(Y)-H(Y|X) = H(X) + H(Y) - H(X,Y) = ...
There are many ways to estimate
MIeither in Matlab, R or even by hand with toy example. However, I couldn't find a single reference mentioning both measures and the differences between them.
My calculation (and understanding) of
MIfor discrete features returns the expected results and match existing
MIf-ns (in R, Matlab). However, Weka has a f-n
InfoGainAttributeEval: http://weka.sourceforge.net/doc.dev/weka/attributeSelection/InfoGainAttributeEval.html, which produces
IGmeasure that is not equal to
MIfor the same data.
There are some similar questions in SE forums and all suggest equivalence of IG and MI (e.g. Information gain, mutual information and related measures). In that case - how IG is calculated in Weka and why it doesn't match MI?
Weka's IG is available from http://featureselection.asu.edu/documentation/infogain.htm in the context of feature selection
Please see accepted answer below. There is no indication that
IG have different meaning in context of information theory - they are the same measures.
Regarding WEKA and different results. I have used Feature Selection package, which always assumed continuous attributes and therefore Weka's discretization was applied before computing Information Gain. Running IG directly from WEKA without discretization, produced the result equivalent to MI.