2
$\begingroup$

I am trying to use sequentialfs to do some feature selection in matlab. I have huge dimensional data of 22215 features. When I tried to use sequentialfs with svm as classifier so that it selects the best subset of features, it just keeps on running, probably its because of the huge number of dimensions. However, weka does the same thing very quickly. Weka has wrapper filters and it does it so quickly. What sort of heuristic does it use? Even though I tried with 5000 features in weka since it was not taking 22215 features, it gave me results quickly with wrapper filters. What should I do with sequentialfs in matlab

This is the command I am using in matlab

c = cvpartition(yS1,'k',12);
opts = statset('display','iter');
[fs, history] = sequentialfs(@SVM_class_fun, X, yS1,'cv', c, 'options', opts);

SVM_class_fun

function err = SVM_class_fun(xTrain, yTrain, xTest, yTest)
  model = svmtrain(xTrain, yTrain);
  err = sum(svmclassify(model, xTest) ~= yTest);
end

Here the dimension of X is 100x22215 where I have 100 examples each of dimension 22215.

$\endgroup$
  • $\begingroup$ Which classifier are using in the wrapper FS of WEKA? Probably, WEKA is not using SVM which takes time to train. $\endgroup$ – soufanom Mar 23 '13 at 11:13
  • $\begingroup$ @soufanom. I am not planning to use weka. I need to do that in matlab. Sequentialfs didn't work in matlab. Am I doing something wrong or it's just not possible using sequentialfs. I need to use some heuristic to prevent searching for all possible features, I gues $\endgroup$ – user34790 Mar 23 '13 at 13:56
  • $\begingroup$ Are you using the default setting of sequentialfs in Matlab including 'direction' option? It might be more appropriate to share your Matlab command. $\endgroup$ – soufanom Mar 23 '13 at 14:08
  • $\begingroup$ @soufanom. I have shared the code. I am currently using SVM, but I am fine with any other classifier. $\endgroup$ – user34790 Mar 23 '13 at 14:23
1
$\begingroup$

Can you not make MatLab call the WEKA and pass data off to it?

For very high dimensional data I find the work of Eugene Tuv to be very useful. Random forests of Gradient Boosted Trees can whittle 100k rows and 100k columns to something useful in a very short time.

http://www.stanford.edu/class/ee392m/Lecture3Tuv.pdf
http://www.journalogy.net/Publication/6491785/feature-selection-with-ensembles-artificial-variables-and-redundancyelimination

$\endgroup$
0
$\begingroup$

That's a lot of features to input into a model. Have you thought about redundancy of information within the features? I would suggest dimensional reduction using e.g. PCA or non-linear manifold learning (diffusion maps, Laplacian eigenmaps, locally preserving projections, Sammon mapping, or even SOM, etc.). Would not some of your features also be noisy with little informativeness?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.