Evaluation of final model in feature selection with nested cross-validation

Question

I am doing feature selection with wrapper method on microarray datasets. I have read several papers and answers here about cross-validation (CV) evaluation on feature selection. Especially the answers here, here and here are very useful. I know it is more robust to use nested CV, so I implemented it with 10-fold CV manner. However there are some points which are unclear for me.

What I have done so far is: I divided samples with stratified CV to train and test samples. On every loop of outer CV I performed feature selection with inner CV on training data. So there is a training data with selected features and test data with selected features. The classifier is trained with training data and tested on test data. The errors on every loop are averaged at the end and this is the error rate of the method I used.

Questions:

1. Should I perform this nested CV for different seeds? If yes for how many seeds? or should I perform this for only one time?

   For the final model I performed the same procedures which I performed on one loop of outer CV, but this time with all dataset. So this part hasn't any relation with nested CV. Feature selection processed over again on all dataset, independent from feature sets that are selected in nested CV. Also no train or test samples.Is it correct?

2. How should I evaluate performance of the final model? I applied 100 times CV with final model on dataset with selected features and then averaged these. Is it a true approach?

3. How I can report the accuracy and selected features? Should I perform the above procedures for different seeds (for ex. 10 different seeds) and then say that for ex:

   • for seed1: 6 selected features, %81 general method accuracy(nested CV), %89 with 100 times CV accuracy.
   • for seed2: 8 selected features, %82 general method accuracy(nested CV), %90 with 100 times CV accuracy.

4. I saw that I can give the best and average accuracy of the 10-foldCV on some optimal selected gene subsets which program selected at final model. If it is appropriate to use 100 times CV for final model evaluation, which 10-foldCV of these hundred ones?

Answer 1

I am facing similar problems so I'll try to give some answers to the best of my knowledge. This source gives some insight and references.

1. Yes, it is best practice to repeat both CV loops with different random initializations. Otherwise you may obtain overly optimistic/pessimistic results from lucky/unlucky splits. I would train the final model by first identifying hyperparameters (including the ones of the feature selection wrapper) via the inner CV loop applied to the whole dataset (no training and test set), than utilize these parameters to train the final model on the whole dataset without CV.

2. No, you cannot evaluate the final model unless you have a separate test set which has not been used by any means before. The performance from nested CV is your estimate of its performance. Doing as you suggested should give you a biased estimate.

3. You may create bootstrapped samples from your data and create the final model for each of these bootstrap datasets. The ratio how often a feature has been selected in the final model would give you an estimate how likely it is to be an important predictor. You may combine this with model-dependent variable importances, e.g., you could provide mean and standard deviation of the bootstrap distributions of model coefficients in case you use a linear model.