4
$\begingroup$

I am reading this paper: "Automated MR image classification in temporal lobe epilepsy", by Focke et al. NeuroImage, 2012.

The authors use support vector machines to classify subjects between healthy controls and patients with epilepsy using magnetic resonance images using leave-one-out cross validation.

Every image provides features in the order of hundreds of thousands. However there are only 38 patients and 22 controls (observations).

In your opinion, is there any problem/disadvantage with this design?

In a situation like this, I would use feature selection and/or dimensionality reduction before classification. It seems intuitive to get rid of uninformative features first.

However, I understand that in the boundary optimization problem associated with SVMs, non informative dimensions are be assigned a low weight, even a zero coefficient. Then, is there then any argument for feature selection prior to SVM training other than facilitate convergence of the optimization problem?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Generally, when doing classification in $p\gg n$ conditions you have a non-negligible chance that some combination of irrelevant attributes will be well correlated with the decision by pure chance, thus yielding deceivingly good classification accuracy and hiding real associations, if any are actually in the data.

Greedy, non-nested feature selection will likely make things worse because real correlations are usually weaker than best spurious ones, and thus will be distilled out; on the same time dimension reduction will further improve the reported accuracy, reaffirming the false impression that everything is OK (this is called overfitting by feature selection).

In other words, doing feature selection has a fair amount of significant benefits (some insight in a problem, better accuracy, simpler training), but may be very dangerous -- thus, well, when there is not enough training samples to do a full-blown validation it is certainly better not to do feature selection than to do it wrong.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Say that you have feature vectors of size 1M, where feature values are taken from a known distribution (e.g. normal) and you assign random labels to each vector. If I understand correctly, what you are saying is that you will always find "good features" in this circumstance. However whatever features you select in your training set, may not be "good" at all when you have another 1M random feature vector presented for testing (LOOCV again) right? $\endgroup$
    – Diego
    Feb 24, 2015 at 3:44
  • $\begingroup$ btw, I agree than doing feature selection on your entire dataset (what you call greedy, non-nested I presume), is not the way to go. As you say, there is a good chance that you will always find good random features when p>>n. So what I usually do is to sub-sample the training set, (N-folds), run feature selection on the folds, and then come up with a heuristic to determine the definitive feature set. $\endgroup$
    – Diego
    Feb 24, 2015 at 3:47
  • $\begingroup$ Sure. Still this "stochastic agreement" may also fail; if only spurious features are selected, they will be likely different each time and there will be no consensus. This is why FS should be all relevant as defined in this Nilsson et al paper. $\endgroup$
    – user88
    Feb 24, 2015 at 11:10
  • $\begingroup$ As a follow-up... In my particular experiment I am discarding features where there is no consensus between the selection folds. Can this in your opinion, contribute to the elimination of spurious relationships? I mean, I am only keeping those features that keep appearing, regardless of the fold where feature selection was executed. And, thanks for the reference I will check it out. $\endgroup$
    – Diego
    Feb 24, 2015 at 16:21
  • 1
    $\begingroup$ Well, obviously with a finite sample you can never be sure that some association is not spurious, but IMHO the fact that a feature keeps appearing in such a procedure is pretty much the best confirmation you can get. And still there is an issue whether something is missing (; You can also check out my paper where something like this is used to benchmark few RF-based feature selectors. $\endgroup$
    – user88
    Feb 25, 2015 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.