I am dealing with a typical $p > n$ problem in the medical field. (typically $p \approx 3700$ and $n \approx 100$ ). The dependent variable is binary (healthy/sick) and features are continuous variables representing intensities of a large set of bio-markers. The bio-markers (i.e. features) are extracted from samples using a feature selection algorithm developed internally. By the nature of the algorithm, the extracted features are dependent on the data set.
Goal is to extract features which are relevant for the outcome of the conditional variable (healthy/sick).
Process: Since we are dealing with little sample sizes, we suggest to use cross validation for the feature selection, rather than applying the algorithm to the whole set, as follows:
- Split original data into testing (10%)/training(90%) data sets.
- Split training data set 10 times into 10 folds (CV).
- In each step of CV apply feature selection algorithm, followed by a non-parametric U-test and sort by FDR adjusted p-values.
- In each step select features with FDR adjusted p-value $\leq 0.05$.
- After CV is finished, collect features from every CV step into one final data set and order them by the number appearances over all steps of CV.
- Select top $k$ (e.g. $k = 10$) features from the final list.
- Develop a classifier (e.g. SVM) by training the selected features from step 6 on the complete training data set.
- Test predictive performance of selected features + model by measuring accuracy on the testing data from step 1.
Does the above procedur make sense? Is there any reference in literature for only selecting features using CV (steps 2 - 6)?
Following https://www.sciencedirect.com/science/article/pii/S0933365715001426 we do not want to train any classifier during the steps in CV. However, if we would use the final list of features and fed it back into CV to train a classier at every step, would that not also introduce bias?
Edit In order to avoid lengthy comments below, the question boils down to: Can CV be used purely for feature selection (reduction) as in steps 3-6?