Based on an excellent and very useful discussion on a previous question i posted (How can I use synthetic data to validate my classification model?), i would like to ask a very specific question about the rfe function of the R package caret which performs recursive feature elimination:
in detail, based on the above link and the discussion regarding the potential selection-bias and overfitting concerning the recursive feature elimination, does the actual function of rfe performs the external validation of the feature selection as illustrated in the Algorithm 2 in section 18.2 of the caret tutorial ? [http://topepo.github.io/caret/recursive-feature-elimination.html]
Of course, It is mentioned below of the above section that:
"The resampling-based Algorithm 2 is in the rfe function. Given the potential selection bias issues, this document focuses on rfe.."
Overall, in order to summarize my question and if i have understood the methodology behind the rfe implemented in caret:
rfe indeed performs the external resampling for validation of the selected features, and it is valid for just using a data set of small sample size as a total training set/input set ? with the final goal of possibly finding common selected features in different random seeds ? (not the repeats of cross-validation in the rfe function)? In other words, if after 10 different random seeds-with 10-fold repeated cross-validation/5 repeats-, some features appeared constantly in all iterations, these would be suggested as features with "increased role in the system under evaluation" ?
A small code chunk example:
library(caret)
library(foreach)
ctrl <- rfeControl(functions = rfFuncs,method = "repeatedcv", repeats = 5, verbose = FALSE)
feature.list <- foreach(i=1:20)%do% {
set.seed(i)
rfProfile <- rfe(x,y=,sizes = 50, rfeControl = ctrl) #arbitary set size
final.subset <-predictors(rfProfile)
}
stable.hub.signature <- Reduce(intersect,feature.list)
Thus, to conclude in the above code my first question is correct about the external resampling validation ?
And generally this kind of feature selection is biased when using these features to train the same dataset that was used for the feature selection, correct ?
