1
$\begingroup$

Based on an excellent and very useful discussion on a previous question i posted (How can I use synthetic data to validate my classification model?), i would like to ask a very specific question about the rfe function of the R package caret which performs recursive feature elimination:

in detail, based on the above link and the discussion regarding the potential selection-bias and overfitting concerning the recursive feature elimination, does the actual function of rfe performs the external validation of the feature selection as illustrated in the Algorithm 2 in section 18.2 of the caret tutorial ? [http://topepo.github.io/caret/recursive-feature-elimination.html]

Of course, It is mentioned below of the above section that:

"The resampling-based Algorithm 2 is in the rfe function. Given the potential selection bias issues, this document focuses on rfe.."

Overall, in order to summarize my question and if i have understood the methodology behind the rfe implemented in caret:

rfe indeed performs the external resampling for validation of the selected features, and it is valid for just using a data set of small sample size as a total training set/input set ? with the final goal of possibly finding common selected features in different random seeds ? (not the repeats of cross-validation in the rfe function)? In other words, if after 10 different random seeds-with 10-fold repeated cross-validation/5 repeats-, some features appeared constantly in all iterations, these would be suggested as features with "increased role in the system under evaluation" ?

A small code chunk example:

library(caret)

library(foreach)

ctrl <- rfeControl(functions = rfFuncs,method = "repeatedcv", repeats = 5, verbose = FALSE)

feature.list <- foreach(i=1:20)%do% {

set.seed(i)

rfProfile <- rfe(x,y=,sizes = 50, rfeControl = ctrl) #arbitary set size

final.subset <-predictors(rfProfile)

}

stable.hub.signature <- Reduce(intersect,feature.list)

Thus, to conclude in the above code my first question is correct about the external resampling validation ?

And generally this kind of feature selection is biased when using these features to train the same dataset that was used for the feature selection, correct ?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.