Info about dataset: 5 classes, 200 trials, 100 features.
(I know about the trial to feature ratio being very low, but can not avoid this here and still got well enough classification results.)
Within a ten-fold cross-validation a SVM classifier was trained for each fold (linear nu-svm, optimal nu was identified by inner ten-fold cross-validation). To handle the multiclass problem the libsvm one-vs-one approach is used: $10=5*4/2$ binary classifiers are trained and then a majority voting scheme used. If two or more classes get the same maximum number of votes one of these is choosen at random (at this point I modified the libsvm default behaviour).
I wanted to compute feature importance scores. I tryed the following procedure:
- load the model of one fold and the matching test data
- predict the test data and remeber the prediction accuracy as baseline
- repeat the following for every feature 500 times:
- permute one feature and get the prediction accuracy on the altered data
- In the end: for every feature compare the prediction accuracy for the intact data with the ones based on the altered data and calculate the average decrease in accuracy.
Averaging these values over all folds should give me an overall importance score i.e. an overall average decrease in accuracy for every feature. The higher this value the more important the feature.
I thought this idea should work out, but the results of this do not look sensible to me: For more or less every feature this yields an average increase in accuracy when permuting this very feature. At least for me that does not make really much sense. Further more none of these averaged values seems to be significant in the sense, that there is no feature for which more than 95% percent of the accuracies with this feature permuted are lower than the baseline.
Any suggestions how to get some sort of importance scores or how to improve my method?
(In Matlab this is actually really really slow procedure as I implemented it at the moment. :-()