Assume I have two feature selection algorithms, A and B, which are developed based on SVM. I applied these two algorithms on the same dataset, a Liver Cancer dataset (400 features & 150 samples), and they selected two small subsets A(30 features) and B(50 features). The classifier used is the same and is a binary SVM. In order to compare which algorithm is better, I then applied 5-fold cross-validation on both subset A and subset B and obtained their ROC/AUC values by using the LIBSVM ROC tool. The error rate is calculated as the mis-classified rates(cancer/non-cancer). I would like to compare which subset can predict the class(labels) better. I have read several posts on this forum and done some googling. With much information, I am very confused now.If everyone with more knowledge can give me some directions, I sincerely appreciate. Here are my questions.
- Is this kind of comparison even meaningful (i.e valid) ?
- if not meaningful, what should be done to correct it ? Or this is still a open research question? if so, what strategy do people commonly use ?
- If I applied both 5 CV(cross-validation) and LOOCV and the results are inconsistent, (i.e. for 5-CV, subset A has better AUC than B and for LOOCV, subset B has better AUC than A), does it mean that bot 5-CV and LOOCV are too sample sensitive and neither one is better ? Or LOOCV should always be a better measurement than 5 CV ?
Thank you very much for helps.