I have a small dataset of 25 observations with a classification variable (factor 0,1) and 82 features scaled to have values between 0 and 1. I used the rfe()
function of the caret
package to identify the best subset of features (2:30)
in terms of metric="ROC"
(which I think is really AUC).
The best subset (14 features) has a AUC of 0.85
, sensitivity of 0.69
, and specificity of 0.67
. However, while the subset of 2 features has a lower AUC of 0.74
, it has a higher sensitivity of 0.77
and equivalent specificity.
(another example is subset 15: equal AUC to 4 digits with better sensitivity)
I understand that AUC is a global measure of diagnostic accuracy, but I'm not quite sure why I'd select a result with a higher AUC and lower combination of sensitivity and specificity. Why is AUC the better metric for choosing between subsets?
# get data (N=25)
library(RCurl)
x <-
getURL("https://gist.githubusercontent.com/ericpgreen/3acb033240f273e4cbfb83d36156839a/raw/42d62fa52b002fc68392e2771d20f0adc4713377/rfeEx2.csv")
df <- read.csv(text = x)
df <- df[, -1]
df$class <- factor(df$class)
# specify and run rfe
library(mlbench)
library(caret)
set.seed(1)
rfFuncs$summary <- twoClassSummary
control <- rfeControl(functions=rfFuncs,
method = "LOOCV",
repeats =5,
number = 10,
returnResamp="final",
verbose = TRUE)
trainctrl <- trainControl(classProbs= TRUE,
summaryFunction = twoClassSummary)
result <- rfe(df[, 2:length(df)], # features
df[, 1], # classification
sizes=2:30,
rfeControl=control,
method="svmRadial",
metric = "ROC",
trControl = trainctrl)
result
Here's the result:
Recursive feature selection
Outer resampling method: Leave-One-Out Cross-Validation
Resampling performance over subset size:
Variables ROC Sens Spec Selected
2 0.7372 0.7692 0.6667
3 0.7308 0.6923 0.6667
4 0.7628 0.6923 0.5000
5 0.7372 0.6923 0.4167
6 0.7179 0.6923 0.5000
7 0.7628 0.6923 0.6667
8 0.7821 0.6923 0.6667
9 0.8013 0.6923 0.6667
10 0.8077 0.6923 0.5833
11 0.8205 0.6923 0.6667
12 0.8397 0.6923 0.5000
13 0.8333 0.6923 0.5833
14 0.8462 0.6923 0.6667 *
15 0.8462 0.7692 0.6667
16 0.8205 0.6923 0.6667
17 0.8237 0.6923 0.6667
18 0.8269 0.6923 0.6667
19 0.8269 0.6923 0.6667
20 0.8269 0.6923 0.5000
21 0.8077 0.6923 0.5000
22 0.8205 0.6923 0.5833
23 0.8333 0.7692 0.5833
24 0.8109 0.6923 0.5833
25 0.8301 0.6923 0.5833
26 0.8429 0.7692 0.6667
27 0.8205 0.7692 0.6667
28 0.8205 0.6923 0.6667
29 0.8205 0.7692 0.6667
30 0.8397 0.7692 0.6667
82 0.8013 0.6923 0.6667
The top 5 variables (out of 14):
v9, v69, v41, v4, v30