Feature selection : Mutual information between 2 features or between feature and target? I am aware that Mutual Information (MI) is one solution for feature reduction techniques.
Consider a binary model with feature vector X = (x1, x2,...,xn) and target y in (0,1)
My question is : MI seems to be done on against the target MI(xi, y) and keep those xi that have high MI with y. 
Why don't we perform MI on pair of features themselves MI(xi, xk) and remove one of those in the pair that has high Mutual Information ? The name Mutual Information seems to suggest MI is done between features (independent variables), not between feature (independent var) and target(dependent var)
Joint Mutual Information 
Mutual Information Wiki
Any difference in your answer if y is not binary ?
 A: Mutual Information is a so called 'filter' method of feature selection.  Filter methods examine the relationship of individual independent variables to the dependent variable using statistical methods (usually linear regression F for regression, Chi-Squared for categorical).  Mutual information is considered a more robust method of filter method feature selection as it is predicated on joint probability.  In other words, where a linear regression F will identify only a linear relationship between independent and dependent variables, mutual information will pick up both linear and non-linear relationships between the variables.
The type of feature selection that involves multiple independent variables and their relationship are known as wrapper methods.  These include forward step methods and backward step methods (e.g. Recursive Feature Elimination).  These methods involve fitting groups of variables to an algorithm and determining which combinations of independent variables perform best.  Forward step methods start with one independent variable and then greedily add another and another testing for the best results.  Backward step methods start with all independent variables and eliminated the one that provides the least contribution and then repeats.
