I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers here, here and here are very useful. I know it is more robust to use nested CV, so I implemented it with 10-fold CV manner. However there are some points which are unclear for me.
What I have done so far is: I divided samples with stratified CV to train and test samples. On every loop of outer CV I performed feature selection with inner CV on training data. So there is a training data with selected features and test data with selected features. The classifier is trained with training data and tested on test data. The errors on every loop are averaged at the end and this is the error rate of the method I used.
Questions:
Should I perform this nested CV for different seeds? If yes for how many seeds? or should I perform this for only one time?
For the final model I performed the same procedures which I performed on one loop of outer CV, but this time with all dataset. So this part hasn't any relation with nested CV. Feature selection processed over again on all dataset, independent from feature sets that are selected in nested CV. Also no train or test samples.Is it correct?
How should I evaluate performance of the final model? I applied 100 times CV with final model on dataset with selected features and then averaged these. Is it a true approach?
How I can report the accuracy and selected features? Should I perform the above procedures for different seeds (for ex. 10 different seeds) and then say that for ex:
- for seed1: 6 selected features, %81 general method accuracy(nested CV), %89 with 100 times CV accuracy.
- for seed2: 8 selected features, %82 general method accuracy(nested CV), %90 with 100 times CV accuracy.
I saw that I can give the best and average accuracy of the 10-foldCV on some optimal selected gene subsets which program selected at final model. If it is appropriate to use 100 times CV for final model evaluation, which 10-foldCV of these hundred ones?