Issues with feature selection in matlab

Question

I am trying to use sequentialfs to do some feature selection in matlab. I have huge dimensional data of 22215 features. When I tried to use sequentialfs with svm as classifier so that it selects the best subset of features, it just keeps on running, probably its because of the huge number of dimensions. However, weka does the same thing very quickly. Weka has wrapper filters and it does it so quickly. What sort of heuristic does it use? Even though I tried with 5000 features in weka since it was not taking 22215 features, it gave me results quickly with wrapper filters. What should I do with sequentialfs in matlab

This is the command I am using in matlab

c = cvpartition(yS1,'k',12);
opts = statset('display','iter');
[fs, history] = sequentialfs(@SVM_class_fun, X, yS1,'cv', c, 'options', opts);

SVM_class_fun

function err = SVM_class_fun(xTrain, yTrain, xTest, yTest)
  model = svmtrain(xTrain, yTrain);
  err = sum(svmclassify(model, xTest) ~= yTest);
end

Here the dimension of X is 100x22215 where I have 100 examples each of dimension 22215.

Answer 1

Can you not make MatLab call the WEKA and pass data off to it?

For very high dimensional data I find the work of Eugene Tuv to be very useful. Random forests of Gradient Boosted Trees can whittle 100k rows and 100k columns to something useful in a very short time.

http://www.stanford.edu/class/ee392m/Lecture3Tuv.pdf
http://www.journalogy.net/Publication/6491785/feature-selection-with-ensembles-artificial-variables-and-redundancyelimination

Answer 2

That's a lot of features to input into a model. Have you thought about redundancy of information within the features? I would suggest dimensional reduction using e.g. PCA or non-linear manifold learning (diffusion maps, Laplacian eigenmaps, locally preserving projections, Sammon mapping, or even SOM, etc.). Would not some of your features also be noisy with little informativeness?

Answer 3

The problem is that you have a very large number of features as compared to that of the observations. There might be many redundant features, you should first check for that, and remove the features based on the rankings obtained. You can compare the correlation among the features (this should be minimised) as well as between the feature and label (this should be maximised), this will provide a clear idea of which are relevant features and which can be got rid of. (Refer to minimum redundancy maximum relevance algorithm for this - Ranking of features using MRMR algorithm)

After reducing the number of features, apply sequential feature selection (sequentialfs() function in MATLAB, here you can avoid SVM and choose LDA to reduce the computational time further).