Based on an excellent and very useful discussion on a previous question i posted (How can I use synthetic data to validate my classification model?), i would like to ask a very specific question about the rfe function of the R package caret which performs recursive feature elimination:

in detail, based on the above link and the discussion regarding the potential selection-bias and overfitting concerning the recursive feature elimination, does the actual function of rfe performs the external validation of the feature selection as illustrated in the Algorithm 2 in section 18.2 of the caret tutorial ? [http://topepo.github.io/caret/recursive-feature-elimination.html]

Of course, It is mentioned below of the above section that:

"The resampling-based Algorithm 2 is in the rfe function. Given the potential selection bias issues, this document focuses on rfe.."

Overall, in order to summarize my question and if i have understood the methodology behind the rfe implemented in caret:

rfe indeed performs the external resampling for validation of the selected features, and it is valid for just using a data set of small sample size as a total training set/input set ? with the final goal of possibly finding common selected features in different random seeds ? (not the repeats of cross-validation in the rfe function)? In other words, if after 10 different random seeds-with 10-fold repeated cross-validation/5 repeats-, some features appeared constantly in all iterations, these would be suggested as features with "increased role in the system under evaluation" ?

A small code chunk example:

library(caret)

library(foreach)

ctrl <- rfeControl(functions = rfFuncs,method = "repeatedcv", repeats = 5, verbose = FALSE)

feature.list <- foreach(i=1:20)%do% {

set.seed(i)

rfProfile <- rfe(x,y=,sizes = 50, rfeControl = ctrl) #arbitary set size

final.subset <-predictors(rfProfile)

}

stable.hub.signature <- Reduce(intersect,feature.list)

Thus, to conclude in the above code my first question is correct about the external resampling validation ?

And generally this kind of feature selection is biased when using these features to train the same dataset that was used for the feature selection, correct ?

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.