3
$\begingroup$

Assume I have two feature selection algorithms, A and B, which are developed based on SVM. I applied these two algorithms on the same dataset, a Liver Cancer dataset (400 features & 150 samples), and they selected two small subsets A(30 features) and B(50 features). The classifier used is the same and is a binary SVM. In order to compare which algorithm is better, I then applied 5-fold cross-validation on both subset A and subset B and obtained their ROC/AUC values by using the LIBSVM ROC tool. The error rate is calculated as the mis-classified rates(cancer/non-cancer). I would like to compare which subset can predict the class(labels) better. I have read several posts on this forum and done some googling. With much information, I am very confused now.If everyone with more knowledge can give me some directions, I sincerely appreciate. Here are my questions.

  • Is this kind of comparison even meaningful (i.e valid) ?
  • if not meaningful, what should be done to correct it ? Or this is still a open research question? if so, what strategy do people commonly use ?
  • If I applied both 5 CV(cross-validation) and LOOCV and the results are inconsistent, (i.e. for 5-CV, subset A has better AUC than B and for LOOCV, subset B has better AUC than A), does it mean that bot 5-CV and LOOCV are too sample sensitive and neither one is better ? Or LOOCV should always be a better measurement than 5 CV ?

Thank you very much for helps.

$\endgroup$
  • 1
    $\begingroup$ Are you cross-validating the feature selection also? If the feature selection was done prior to the cross-validation on the same data set, the results from any cross-validation procedure are most likely optimistic. The feature selection step must be included inside the cross-validation loop. $\endgroup$ – NRH Apr 5 '13 at 10:16
2
+50
$\begingroup$

Your procedure is not valid because you perform your feature selection outside of your cross validation loop. This means the feature selection algorithms are able to selected features which are not optimal only on the training set but on the entire data. This issue is discussed, for example in The Elements of Statistical Learning which can also be found for free online.

If you want to compare the algorithms you can embed each into the cross validation and look at the results and do a statistical test whether one yields significantly better results than the other one. There are bootstrap and parametrical tests for the AUC available. Note however, that selecting one of the two approaches based on this result will incur an upward bias in your performance estimates, since you pick the better of two approaches. If you want to still adequately estimate your overall performance, you can also embed the choice into your cross validation, by using nested cross validation. This is also commonly used to tune model parameters.

You should also think about using Bootstrapping instead of Cross Validation since it makes more efficient use of the data. Frank Harrell has written a lot about that, see either his book on Regression Modeling Strategies or his comments and posts in this forum (he is a member).

Do not use LOOCV for comparing feature selection algorithms. Since the training set will be almost identical for each run, so will be the feature selection. This means you won't be able to adequately capture and assess the variability involved in the feature selection algorithms.

$\endgroup$
  • $\begingroup$ Thank you very much for replying. Would you please explain a little bit further what you mean "If you want to compare the algorithms you can embed each into the cross validation "? How do you embed them into the CV? I am confused. Thanks, $\endgroup$ – Cassie Apr 6 '13 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.