I am using the
caret package to do feature selection with
rfe while training a
knn classifier. I want to tune both the
k parameter and the
size of the variable subset by taking the best value of the metric
ROC from the
twoClassSummary() function. My code to accomplish that is:
caretFuncs$summary <- twoClassSummary knn.rfeC <- rfeControl(functions = caretFuncs, method = "repeatedcv", number=10, repeats=5, verbose = TRUE, returnResamp = "final") knn.grid <- expand.grid(.k=seq(1, 20, 2)) knn.trainC <- trainControl(method = "boot", number=25, verboseIter=TRUE, returnResamp="final", classProbs=T, summaryFunction = twoClassSummary) sizes <- 2:(ncol(train.x)-1) set.seed(96) knnR <- rfe(train.x, train.y, sizes = sizes, rfeControl = knn.rfeC, method="knn", tuneGrid=knn.grid, metric="ROC", trControl=knn.trainC)
rfe finishes, I get the following warning message:
1: In train.default(x, y, ...) : The metric "Accuracy" was not in the result set. Sens will be used instead.
which I gather it means that the
metric argument of
rfe is only working on the subset size selection, but not the
k parameter optimization, even though the
twoClassSummary function is passed to the
summaryFunction argument of
Am I missing something here?
I would appreciate any help you could provide.
The output for
> sessionInfo() R version 2.13.1 (2011-07-08) Platform: i386-pc-mingw32/i386 (32-bit) locale:  LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252  LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C  LC_TIME=English_United Kingdom.1252 attached base packages:  stats graphics grDevices utils datasets methods base other attached packages:  caret_4.92 cluster_1.14.0 reshape_0.8.4 plyr_1.5.2 lattice_0.19-30 loaded via a namespace (and not attached):  grid_2.13.1 tools_2.13.1