Mutual Information based feature selection

Question

Suppose I have a feature matrix $F = [f_1^T,f_2^T,...,f_m^T]$ whereby $f_j^T \in \mathbb R^{n_t \times 1}$ is the $j$th column vector of $F$ ($n_t$ is the number of different events/trials and $m$ is the total number of features per trial). All trials are labelled with classlabel $\omega = \{1,2\}$.

Therefore, in simpler words, each row of $F$ contains the features of a particular trial whose label is known.

My problem is this...I am trying to make use of Mutual Information to select the best features possible. The notation I have shown so far is all from a particular paper and it suggests that I need to calculate the mutual information of each feature $f_j$ with the class label $\omega =\{1,2\}$ as follows:

$I(f_j;w)$ $\forall $ $j=1,2...m$

Does this make sense? If so, does this mean i am computing the mutual information of the first feature of each trial with their corresponding classlabel, of the second feature of each trial with their corresponding classlabel etc...? How many weights will I finally end up with? Because then I would like to sort them in descending order and choose the best k features.

Just in case, a snapshot of the part of the paper I am talking about is this: http://img547.imageshack.us/img547/8531/82104489.png

I wasn't able to attach the whole paper.

Thanks a lot for your help!

Answer 1

You are as such correct, but I would suggest using Weka to do it for you. For example, the following piece of java code will help you choose the attributes by mutual information using Weka.

    Instances trainingInstances; // feature matrix
    int n = ...; // number of features to select
    AttributeSelection attributeSelection = new AttributeSelection();
    InfoGainAttributeEval infoGainAttributeEval = new InfoGainAttributeEval();
    Ranker ranker = new Ranker();
    ranker.setNumToSelect(n);
    attributeSelection.setEvaluator(infoGainAttributeEval);
    attributeSelection.setSearch(ranker);
    attributeSelection.setInputFormat(trainingInstances);
    Instances featureSelected = Filter.useFilter(trainingInstances, attributeSelection);

I am assuming here that you have familiarity using Weka library.

Answer 2

You are correct that this process will produce one weight per feature, its mutual information with the class label. That won't necessarily produce the best feature ranking though, as it will have redundant features (i.e. where two features are very similar, you will pick both of them). If you are using MIToolbox, all the features need to be discretised, as it only measures the discrete mutual information (the continuous mutual information can take negative values and is more difficult to calculate).

We also wrote a feature selection package based on MIToolbox, called FEAST, which has implementations of many mutual information based feature selection algorithms.