Feature Selection: Information Gain VS Mutual Information - Cross Validated most recent 30 from stats.stackexchange.com 2019-06-26T08:07:44Z https://stats.stackexchange.com/feeds/question/116679 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/116679 4 Feature Selection: Information Gain VS Mutual Information Oleg Shirokikh https://stats.stackexchange.com/users/30331 2014-09-24T22:21:32Z 2014-10-20T01:20:03Z <p>Setting: Multi-class classification problem with discrete nominal features.</p> <p>There are many references mentioning the use of <code>IG</code>(Information Gain) and <code>MI</code> (Mutual Information) as measure of feature relevancy for filter-based feature selection. However, from the information-theoretic viewpoint it's not completely clear to me what is the difference between these two (and if there is any):</p> <pre><code>IG(X,Y) ?=? MI(X,Y) = H(X)-H(X|Y) = H(Y)-H(Y|X) = H(X) + H(Y) - H(X,Y) = ... </code></pre> <p>Notes:</p> <ul> <li><p>There are many ways to estimate <code>MI</code> either in Matlab, R or even by hand with toy example. However, I couldn't find a single reference mentioning <strong>both</strong> measures and the differences between them.</p></li> <li><p>My calculation (and understanding) of <code>MI</code> for discrete features returns the expected results and match existing <code>MI</code> f-ns (in R, Matlab). However, Weka has a f-n <code>InfoGainAttributeEval</code>: <a href="http://weka.sourceforge.net/doc.dev/weka/attributeSelection/InfoGainAttributeEval.html" rel="nofollow">http://weka.sourceforge.net/doc.dev/weka/attributeSelection/InfoGainAttributeEval.html</a>, which produces <code>IG</code> measure that is not equal to <code>MI</code> for the same data.</p></li> <li><p>There are some similar questions in SE forums and all suggest equivalence of IG and MI (e.g. <a href="https://stats.stackexchange.com/questions/13389/information-gain-mutual-information-and-related-measures">Information gain, mutual information and related measures</a>). In that case - how IG is calculated in Weka and why it doesn't match MI?</p></li> <li><p>Weka's IG is available from <a href="http://featureselection.asu.edu/documentation/infogain.htm" rel="nofollow">http://featureselection.asu.edu/documentation/infogain.htm</a> in the context of feature selection</p></li> </ul> <p><strong>ANSWER:</strong></p> <p>Please see accepted answer below. There is no indication that <code>MI</code> and <code>IG</code> have different meaning in context of information theory - they are the same measures.</p> <p>Regarding WEKA and different results. I have used Feature Selection package, which <strong>always</strong> assumed continuous attributes and therefore Weka's discretization was applied before computing Information Gain. Running IG directly from WEKA without discretization, produced the result equivalent to MI.</p> https://stats.stackexchange.com/questions/116679/-/116686#116686 3 Answer by Simone for Feature Selection: Information Gain VS Mutual Information Simone https://stats.stackexchange.com/users/2719 2014-09-24T23:21:53Z 2014-09-24T23:21:53Z <p>They are identically the same.</p> <p>If $X$ is a nominal feature with $k$ different values and $C$ is your target class with $m$ classes.</p> <p>\begin{align} \mbox{MI} &amp;= \sum_{i=1}^k\sum_{j=1}^m P(x_i,c_j)\log \frac{P(x_i,c_j)}{P(x_i)P(c_j)}\\ &amp;=-\sum_{i=1}^k\sum_{j=1}^m P(x_i,c_j)\log P(c_j) + \sum_{i=1}^k\sum_{j=1}^m P(c_j|x_i)P(x_i)\log P(c_j|x_i)\\ &amp;= -\sum_{j=1}^m P(c_j)\log P(c_j) + \sum_{i=1}^kP(x_i)\sum_{j=1}^m P(c_j|x_i)\log P(c_j|x_i)\\ &amp;= H(C) - H(C|X) = \mbox{IG} \end{align}</p> <p>This is a case of inconsistent naming. You might want to have a look at this <a href="https://stats.stackexchange.com/questions/13389/information-gain-mutual-information-and-related-measures">question</a> too. </p>