recursive feature elimination: why select subset based on AUC vs sensitivity/specificity

I have a small dataset of 25 observations with a classification variable (factor 0,1) and 82 features scaled to have values between 0 and 1. I used the rfe() function of the caret package to identify the best subset of features (2:30) in terms of metric="ROC" (which I think is really AUC).

The best subset (14 features) has a AUC of 0.85, sensitivity of 0.69, and specificity of 0.67. However, while the subset of 2 features has a lower AUC of 0.74, it has a higher sensitivity of 0.77 and equivalent specificity.

(another example is subset 15: equal AUC to 4 digits with better sensitivity)

I understand that AUC is a global measure of diagnostic accuracy, but I'm not quite sure why I'd select a result with a higher AUC and lower combination of sensitivity and specificity. Why is AUC the better metric for choosing between subsets?

# get data (N=25)
library(RCurl)
x <-
df$class <- factor(df$class)
rfFuncs$summary <- twoClassSummary control <- rfeControl(functions=rfFuncs, method = "LOOCV", repeats =5, number = 10, returnResamp="final", verbose = TRUE) trainctrl <- trainControl(classProbs= TRUE, summaryFunction = twoClassSummary) result <- rfe(df[, 2:length(df)], # features df[, 1], # classification sizes=2:30, rfeControl=control, method="svmRadial", metric = "ROC", trControl = trainctrl) result  Here's the result: Recursive feature selection Outer resampling method: Leave-One-Out Cross-Validation Resampling performance over subset size: Variables ROC Sens Spec Selected 2 0.7372 0.7692 0.6667 3 0.7308 0.6923 0.6667 4 0.7628 0.6923 0.5000 5 0.7372 0.6923 0.4167 6 0.7179 0.6923 0.5000 7 0.7628 0.6923 0.6667 8 0.7821 0.6923 0.6667 9 0.8013 0.6923 0.6667 10 0.8077 0.6923 0.5833 11 0.8205 0.6923 0.6667 12 0.8397 0.6923 0.5000 13 0.8333 0.6923 0.5833 14 0.8462 0.6923 0.6667 * 15 0.8462 0.7692 0.6667 16 0.8205 0.6923 0.6667 17 0.8237 0.6923 0.6667 18 0.8269 0.6923 0.6667 19 0.8269 0.6923 0.6667 20 0.8269 0.6923 0.5000 21 0.8077 0.6923 0.5000 22 0.8205 0.6923 0.5833 23 0.8333 0.7692 0.5833 24 0.8109 0.6923 0.5833 25 0.8301 0.6923 0.5833 26 0.8429 0.7692 0.6667 27 0.8205 0.7692 0.6667 28 0.8205 0.6923 0.6667 29 0.8205 0.7692 0.6667 30 0.8397 0.7692 0.6667 82 0.8013 0.6923 0.6667 The top 5 variables (out of 14): v9, v69, v41, v4, v30  1 Answer Disclaimer: I do not generally endorse the use automatic feature selection algorithms. They are prone to overfitting, emphasising nonsensical associations, over-reliance on a particular metric used and commonly present a good tool for senseless data analysis. They are probably useful for variable importance at some settings where data collection cost is an issue... :) To your question now: The choice of AUC-ROC is just a standard choice and you should not read too much into it. AUC-ROC can be seen as a metric that does not depend on particular probability thresholds in the case of probabilistic classification. In addition it has a nice association with the Mann-Whitney U-test (ie. can be easily interpreted as the probability that the classifier will assign a higher probability to a randomly chosen positive item than a randomly chosen negative item) and it is not too affected by imbalanced data. For these reasons it is the established vanilla ice-cream of classification metrics. As you correctly assess maximising AUC-ROC does not guarantee that we will get the better combination of sensitivity and specificity or any other particular metric. (If you are interested in the combination of the two you might want to consider using F-score) In most applications defining a correct metric is half the battle. For a probabilistic classifier, I would suggest you look into using a proper scoring rule like the Brier score. Please note that accuracy is usually not a good metric because it is an improper rule (ie. maximising it does not guarantee we are as close as possible to the true probability distributions of the data). Additional notes particular to the use case present: 1. For Support Vector Machines AUC-ROC is a bit fuzzy as a concept. SVMs natively do hard class-assignment and have to rely to isotonic regression or Platt scaling approaches to generate probability distributions over classes. This adds an additional layer of complexity. 2. At first instance, I could suggest using a "standard" regularisation approach like Elastic-net or Least-angle regression (to directly control for the$n \ll p$scenario you outline. The techniques will do the "variable selection" internally through regularisation. 3. The sensitivity and specificity values you see in the output of rfe are related to a$0.5$probability threshold. It is plausible you can get better performance using another threshold. 4. AUC-ROC is probably better than just using sensitivity and specificity. Both sensitivity and specificity are improper scoring rules that are hard to properly optimise for. For example, correctly increasing the probability of an item belonging to particular class does not increase neither sensitivity or specificity unless the increase passes an arbitrary threshold (usually$0.5\$). This correct change in the estimated probability will be directly reflected on the AUC-ROC value.
5. ROC, ie. Receiver operating characteristic is used instead of simply AUC Area-Under-the-Curve because AUC can be easily defined for many other metrics too. A common one is AUC-PR, the Area-Under-the-Curve for the Precision Recall curve - a quite popular metric for imbalanced learning (eg. see Davis and Goadrich's The Relationship Between Precision-Recall and ROC Curves for more on this.)