I am using the caret package to do feature selection with rfe while training a knn classifier. I want to tune both the k parameter and the size of the variable subset by taking the best value of the metric ROC from the twoClassSummary() function. My code to accomplish that is:

caretFuncs$summary <- twoClassSummary
knn.rfeC <- rfeControl(functions = caretFuncs, method = "repeatedcv", number=10, repeats=5, 
                      verbose = TRUE, returnResamp = "final")
knn.grid <- expand.grid(.k=seq(1, 20, 2))
knn.trainC <- trainControl(method = "boot", number=25, verboseIter=TRUE, returnResamp="final", 
                      classProbs=T, summaryFunction = twoClassSummary)
sizes <- 2:(ncol(train.x)-1)
knnR <- rfe(train.x, train.y, sizes = sizes, rfeControl = knn.rfeC, method="knn",
           tuneGrid=knn.grid, metric="ROC", trControl=knn.trainC)

However, when rfe finishes, I get the following warning message:

1: In train.default(x, y, ...) :
The metric "Accuracy" was not in the result set. Sens will be used instead.

which I gather it means that the metric argument of rfe is only working on the subset size selection, but not the k parameter optimization, even though the twoClassSummary function is passed to the summaryFunction argument of trainControl.

Am I missing something here?

I would appreciate any help you could provide.

EDIT: The output for sessionInfo() is:

> sessionInfo()
R version 2.13.1 (2011-07-08)
Platform: i386-pc-mingw32/i386 (32-bit)

[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] caret_4.92      cluster_1.14.0  reshape_0.8.4   plyr_1.5.2      lattice_0.19-30

loaded via a namespace (and not attached):
[1] grid_2.13.1  tools_2.13.1
  • $\begingroup$ Can you send the results of sessionInfo()? $\endgroup$ – topepo Nov 6 '13 at 17:00
  • $\begingroup$ @topepo: Hi. I have updated my question with the sessionInfo() results. $\endgroup$ – qkn101 Nov 7 '13 at 8:43
  • $\begingroup$ Was the session info any help, @topepo ? Should I provide more info? $\endgroup$ – qkn101 Nov 20 '13 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.