Skip to main content
added sentence about dependency
Source Link

Mutual Information (sometimes called Information Gain) is a decrease in information entropy of one random variable when another is known. It measures dependency (not only linear) between random variables. It has formula: $$ I(X,Y) = \sum\limits_{x, y}P(X=x, Y=y)\ln\left(\frac{P(X=x,Y=y)}{P(X=x)P(Y=y)}\right) $$ It can be used as feature utility metric.

Information Value measures predictive power of a feature and is given by formula (sometimes modified with binning): $$ IV(X, Y) = \sum_{x}(P(X=x|Y=1) - P(X=x|Y=0))\ln\left(\frac{P(X=x|Y=1)}{P(X=x|Y=1)}\right) $$

Although similar, Information Value and Information Gain (Mutual Information) are different things. Both are used in univariate feature selection. Both can be expressed in terms of Kullback-Leibler divergence (for calculation for IV check this answer).

Mutual Information (sometimes called Information Gain) is a decrease in information entropy of one random variable when another is known. It has formula: $$ I(X,Y) = \sum\limits_{x, y}P(X=x, Y=y)\ln\left(\frac{P(X=x,Y=y)}{P(X=x)P(Y=y)}\right) $$ It can be used as feature utility metric.

Information Value measures predictive power of a feature and is given by formula (sometimes modified with binning): $$ IV(X, Y) = \sum_{x}(P(X=x|Y=1) - P(X=x|Y=0))\ln\left(\frac{P(X=x|Y=1)}{P(X=x|Y=1)}\right) $$

Although similar, Information Value and Information Gain (Mutual Information) are different things. Both are used in univariate feature selection. Both can be expressed in terms of Kullback-Leibler divergence (for calculation for IV check this answer).

Mutual Information (sometimes called Information Gain) is a decrease in information entropy of one random variable when another is known. It measures dependency (not only linear) between random variables. It has formula: $$ I(X,Y) = \sum\limits_{x, y}P(X=x, Y=y)\ln\left(\frac{P(X=x,Y=y)}{P(X=x)P(Y=y)}\right) $$ It can be used as feature utility metric.

Information Value measures predictive power of a feature and is given by formula (sometimes modified with binning): $$ IV(X, Y) = \sum_{x}(P(X=x|Y=1) - P(X=x|Y=0))\ln\left(\frac{P(X=x|Y=1)}{P(X=x|Y=1)}\right) $$

Although similar, Information Value and Information Gain (Mutual Information) are different things. Both are used in univariate feature selection. Both can be expressed in terms of Kullback-Leibler divergence (for calculation for IV check this answer).

Source Link

Mutual Information (sometimes called Information Gain) is a decrease in information entropy of one random variable when another is known. It has formula: $$ I(X,Y) = \sum\limits_{x, y}P(X=x, Y=y)\ln\left(\frac{P(X=x,Y=y)}{P(X=x)P(Y=y)}\right) $$ It can be used as feature utility metric.

Information Value measures predictive power of a feature and is given by formula (sometimes modified with binning): $$ IV(X, Y) = \sum_{x}(P(X=x|Y=1) - P(X=x|Y=0))\ln\left(\frac{P(X=x|Y=1)}{P(X=x|Y=1)}\right) $$

Although similar, Information Value and Information Gain (Mutual Information) are different things. Both are used in univariate feature selection. Both can be expressed in terms of Kullback-Leibler divergence (for calculation for IV check this answer).