I am doing feature selection using the command 'rfe' in the caret package (http://caret.r-forge.r-project.org/featureselection.html). This command uses a metric to find the optimal amount of variables and which variables that is. However, I would like to also see the other steps in the feature selection than simply the last one. For instance, I would like to know which variables were the optimal ones if I wanted exactly 10 variables.

My code is the following:

ctrl <- rfeControl(functions = rfFuncs,
                   method = "cv",
                   verbose = FALSE)
subsets <- c(5,10,15,20,25)
lmProfile <- rfe(dat2_X, dat2_Y,
                 sizes = subsets,
                 rfeControl = ctrl)

See lmProfile$variables. It has the ranking metrics for each predictor at each iteration. For example, from ?rfe:


x <- scale(bbbDescr[,-nearZeroVar(bbbDescr)])
x <- x[, -findCorrelation(cor(x), .8)]
x <- as.data.frame(x)

lmProfile <- rfe(x, logBBB,
                 sizes = 10:20,
                 rfeControl = rfeControl(functions = lmFuncs, 
                                         number = 15))

head(lmProfile$variables) has:

Overall            var Variables   Resample
4.930084     vsa_other        71 Resample01
4.696723    slogp_vsa5        71 Resample01
3.877510         pnsa1        71 Resample01
3.649555      vsa_base        71 Resample01
3.586327 frac.cation7.        71 Resample01
3.301325        a_base        71 Resample01

For each resample, there are 71 rows here that are the variables selected for a subset size of 71, 20 rows for the ones selected at 20 etc.



Just to complete Max's answer, here is one way to ensure that you select variables of your choice of subset-size from the output given above. Suppose you want a set-size of 10:

set.size = 10 #you want set-size of 10
lm.vars <- lmProfile$variables 

lm.set <- lm.vars[lm.vars$Variables==set.size,  ] # selects variables of set-size (= 10 here)

The issue is that in different folds, different variables may be selected in the set of 10. So we average the importance/ranking score of all variables in set, and choose the highest ranked 10 variables

#use aggregate to calculate mean ranking score (under column "Overall")
lm.set <- aggregate(lm.set[, c("Overall")], list(lm.set$var), mean)

#order from highest to low, and select first 10:
lm.order <- order(lm.set[, c("x")], decreasing = TRUE)[1:set.size]
lm.set[lm.order, ]

This will give the following output:

                Group.1           x
9              hardness 452.3865743
10                 homo 363.2872843
11                 lumo 283.3329715
20                pnsa1   1.0754635
29             smr_vsa0   1.0078744
13 most_positive_charge   0.9588959
6                 fnsa1   0.9310444
14                o_sp2   0.8874429
1                 a_acc   0.8706394
18         peoe_vsa.5.1   0.8072423    

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.