# Mutual Information based feature selection

Suppose I have a feature matrix $F = [f_1^T,f_2^T,...,f_m^T]$ whereby $f_j^T \in \mathbb R^{n_t \times 1}$ is the $j$th column vector of $F$ ($n_t$ is the number of different events/trials and $m$ is the total number of features per trial). All trials are labelled with classlabel $\omega = \{1,2\}$.

Therefore, in simpler words, each row of $F$ contains the features of a particular trial whose label is known.

My problem is this...I am trying to make use of Mutual Information to select the best features possible. The notation I have shown so far is all from a particular paper and it suggests that I need to calculate the mutual information of each feature $f_j$ with the class label $\omega =\{1,2\}$ as follows:

$I(f_j;w)$ $\forall$ $j=1,2...m$

Does this make sense? If so, does this mean i am computing the mutual information of the first feature of each trial with their corresponding classlabel, of the second feature of each trial with their corresponding classlabel etc...? How many weights will I finally end up with? Because then I would like to sort them in descending order and choose the best k features.

Just in case, a snapshot of the part of the paper I am talking about is this: http://img547.imageshack.us/img547/8531/82104489.png

I wasn't able to attach the whole paper.

Thanks a lot for your help!

• what is the name of this paper ? – Qbik Apr 10 '13 at 16:40

You are as such correct, but I would suggest using Weka to do it for you. For example, the following piece of java code will help you choose the attributes by mutual information using Weka.

    Instances trainingInstances; // feature matrix
int n = ...; // number of features to select
AttributeSelection attributeSelection = new AttributeSelection();
InfoGainAttributeEval infoGainAttributeEval = new InfoGainAttributeEval();
Ranker ranker = new Ranker();
ranker.setNumToSelect(n);
attributeSelection.setEvaluator(infoGainAttributeEval);
attributeSelection.setSearch(ranker);
attributeSelection.setInputFormat(trainingInstances);
Instances featureSelected = Filter.useFilter(trainingInstances, attributeSelection);


I am assuming here that you have familiarity using Weka library.

• Unfortunately I don't have any experience with the Weka library. I'm doing all of this with MATLAB – Lunat1c Mar 24 '13 at 11:01
• Oh, in that case, you can probably try: mathworks.in/matlabcentral/fileexchange/… or cs.man.ac.uk/~pococka4/MIToolbox.html – TenaliRaman Mar 25 '13 at 3:51
• I have computed the algorithm based on the equations from the paper shown in the screenshot of my first post. The only issue is that the MI weights I am getting are all negative. Is that normal? Specifically, I got these values for the 16 features I have: [-45.9936, -51.1898,-48.6486,-62.0466,-69.0063,-70.3399, -70.7139,-69.9357,-69.9412,-68.8998,-68.5802,-67.1621, -65.0743,-54.5790,-57.1689,-55.3879] – Lunat1c Mar 29 '13 at 12:36
• MI is non-negative. Are you sure you haven't missed any sign or something? Since all your weights are negative, it feels like you might missed a minus sign somewhere. – TenaliRaman Mar 29 '13 at 13:03
• I have checked and rechecked and I can't seem to find it. Sorting those in descending order and choosing the corresponding features resulted in a good-ish accuracy. in ascending order the accuracy was horrible. but since you're telling me that MI is non-negative maybe there's something I'm missing – Lunat1c Mar 29 '13 at 15:24

You are correct that this process will produce one weight per feature, its mutual information with the class label. That won't necessarily produce the best feature ranking though, as it will have redundant features (i.e. where two features are very similar, you will pick both of them). If you are using MIToolbox, all the features need to be discretised, as it only measures the discrete mutual information (the continuous mutual information can take negative values and is more difficult to calculate).

We also wrote a feature selection package based on MIToolbox, called FEAST, which has implementations of many mutual information based feature selection algorithms.