I am aware that Mutual Information (MI) is one solution for feature reduction techniques.

Consider a binary model with feature vector X = (x1, x2,...,xn) and target y in (0,1)

My question is : MI seems to be done on against the target MI(xi, y) and keep those xi that have high MI with y.

Why don't we perform MI on pair of features themselves MI(xi, xk) and remove one of those in the pair that has high Mutual Information ? The name Mutual Information seems to suggest MI is done between features (independent variables), not between feature (independent var) and target(dependent var)

Joint Mutual Information

Mutual Information Wiki

Any difference in your answer if y is not binary ?


1 Answer 1


Mutual Information is a so called 'filter' method of feature selection. Filter methods examine the relationship of individual independent variables to the dependent variable using statistical methods (usually linear regression F for regression, Chi-Squared for categorical). Mutual information is considered a more robust method of filter method feature selection as it is predicated on joint probability. In other words, where a linear regression F will identify only a linear relationship between independent and dependent variables, mutual information will pick up both linear and non-linear relationships between the variables.

The type of feature selection that involves multiple independent variables and their relationship are known as wrapper methods. These include forward step methods and backward step methods (e.g. Recursive Feature Elimination). These methods involve fitting groups of variables to an algorithm and determining which combinations of independent variables perform best. Forward step methods start with one independent variable and then greedily add another and another testing for the best results. Backward step methods start with all independent variables and eliminated the one that provides the least contribution and then repeats.

  • $\begingroup$ Thanks. Can you see my updates and point me to resources that states MI should be done between independent var vs. dependent var ? I understand from the name and definition that it's to calculate Mutual Information to remove redundancy between variables, that's what helps the feature selection. Not about MI between feature and target $\endgroup$
    – Kenny
    Nov 14, 2019 at 17:04
  • $\begingroup$ It's actually addressed in the article linked to Joint Mutual Information $\endgroup$ Nov 14, 2019 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.