Is it possible to compare two feature selections algorithms by cross-validations?

Question

Assume I have two feature selection algorithms, A and B, which are developed based on SVM. I applied these two algorithms on the same dataset, a Liver Cancer dataset (400 features & 150 samples), and they selected two small subsets A(30 features) and B(50 features). The classifier used is the same and is a binary SVM. In order to compare which algorithm is better, I then applied 5-fold cross-validation on both subset A and subset B and obtained their ROC/AUC values by using the LIBSVM ROC tool. The error rate is calculated as the mis-classified rates(cancer/non-cancer). I would like to compare which subset can predict the class(labels) better. I have read several posts on this forum and done some googling. With much information, I am very confused now.If everyone with more knowledge can give me some directions, I sincerely appreciate. Here are my questions.

• Is this kind of comparison even meaningful (i.e valid) ?
• if not meaningful, what should be done to correct it ? Or this is still a open research question? if so, what strategy do people commonly use ?
• If I applied both 5 CV(cross-validation) and LOOCV and the results are inconsistent, (i.e. for 5-CV, subset A has better AUC than B and for LOOCV, subset B has better AUC than A), does it mean that bot 5-CV and LOOCV are too sample sensitive and neither one is better ? Or LOOCV should always be a better measurement than 5 CV ?

Thank you very much for helps.

Answer 1

Your procedure is not valid because you perform your feature selection outside of your cross validation loop. This means the feature selection algorithms are able to selected features which are not optimal only on the training set but on the entire data. This issue is discussed, for example in The Elements of Statistical Learning which can also be found for free online.

If you want to compare the algorithms you can embed each into the cross validation and look at the results and do a statistical test whether one yields significantly better results than the other one. There are bootstrap and parametrical tests for the AUC available. Note however, that selecting one of the two approaches based on this result will incur an upward bias in your performance estimates, since you pick the better of two approaches. If you want to still adequately estimate your overall performance, you can also embed the choice into your cross validation, by using nested cross validation. This is also commonly used to tune model parameters.

You should also think about using Bootstrapping instead of Cross Validation since it makes more efficient use of the data. Frank Harrell has written a lot about that, see either his book on Regression Modeling Strategies or his comments and posts in this forum (he is a member).

Do not use LOOCV for comparing feature selection algorithms. Since the training set will be almost identical for each run, so will be the feature selection. This means you won't be able to adequately capture and assess the variability involved in the feature selection algorithms.