recursive feature elimination: why select subset based on AUC vs sensitivity/specificity

Question

I have a small dataset of 25 observations with a classification variable (factor 0,1) and 82 features scaled to have values between 0 and 1. I used the rfe() function of the caret package to identify the best subset of features (2:30) in terms of metric="ROC" (which I think is really AUC).

The best subset (14 features) has a AUC of 0.85, sensitivity of 0.69, and specificity of 0.67. However, while the subset of 2 features has a lower AUC of 0.74, it has a higher sensitivity of 0.77 and equivalent specificity.

(another example is subset 15: equal AUC to 4 digits with better sensitivity)

I understand that AUC is a global measure of diagnostic accuracy, but I'm not quite sure why I'd select a result with a higher AUC and lower combination of sensitivity and specificity. Why is AUC the better metric for choosing between subsets?

# get data (N=25)
  library(RCurl)
  x <-
 getURL("https://gist.githubusercontent.com/ericpgreen/3acb033240f273e4cbfb83d36156839a/raw/42d62fa52b002fc68392e2771d20f0adc4713377/rfeEx2.csv")
  df <- read.csv(text = x)
  df <- df[, -1]
  df$class <- factor(df$class)

# specify and run rfe
  library(mlbench)
  library(caret)
  set.seed(1)
  rfFuncs$summary <- twoClassSummary  
  control <- rfeControl(functions=rfFuncs, 
                        method = "LOOCV",
                        repeats =5, 
                        number = 10,
                        returnResamp="final", 
                        verbose = TRUE)
  trainctrl <- trainControl(classProbs= TRUE,
                            summaryFunction = twoClassSummary)
  result <- rfe(df[, 2:length(df)],         # features
                df[, 1],                    # classification
                sizes=2:30,             
                rfeControl=control,
                method="svmRadial",
                metric = "ROC",
                trControl = trainctrl)
  result

Here's the result:

Recursive feature selection

Outer resampling method: Leave-One-Out Cross-Validation 

Resampling performance over subset size:

 Variables    ROC   Sens   Spec Selected
         2 0.7372 0.7692 0.6667         
         3 0.7308 0.6923 0.6667         
         4 0.7628 0.6923 0.5000         
         5 0.7372 0.6923 0.4167         
         6 0.7179 0.6923 0.5000         
         7 0.7628 0.6923 0.6667         
         8 0.7821 0.6923 0.6667         
         9 0.8013 0.6923 0.6667         
        10 0.8077 0.6923 0.5833         
        11 0.8205 0.6923 0.6667         
        12 0.8397 0.6923 0.5000         
        13 0.8333 0.6923 0.5833         
        14 0.8462 0.6923 0.6667        *
        15 0.8462 0.7692 0.6667         
        16 0.8205 0.6923 0.6667         
        17 0.8237 0.6923 0.6667         
        18 0.8269 0.6923 0.6667         
        19 0.8269 0.6923 0.6667         
        20 0.8269 0.6923 0.5000         
        21 0.8077 0.6923 0.5000         
        22 0.8205 0.6923 0.5833         
        23 0.8333 0.7692 0.5833         
        24 0.8109 0.6923 0.5833         
        25 0.8301 0.6923 0.5833         
        26 0.8429 0.7692 0.6667         
        27 0.8205 0.7692 0.6667         
        28 0.8205 0.6923 0.6667         
        29 0.8205 0.7692 0.6667         
        30 0.8397 0.7692 0.6667         
        82 0.8013 0.6923 0.6667         

The top 5 variables (out of 14):
   v9, v69, v41, v4, v30

Answer 1

Disclaimer: I do not generally endorse the use automatic feature selection algorithms. They are prone to overfitting, emphasising nonsensical associations, over-reliance on a particular metric used and commonly present a good tool for senseless data analysis. They are probably useful for variable importance at some settings where data collection cost is an issue... :) To your question now:

The choice of AUC-ROC is just a standard choice and you should not read too much into it. AUC-ROC can be seen as a metric that does not depend on particular probability thresholds in the case of probabilistic classification. In addition it has a nice association with the Mann-Whitney U-test (ie. can be easily interpreted as the probability that the classifier will assign a higher probability to a randomly chosen positive item than a randomly chosen negative item) and it is not too affected by imbalanced data. For these reasons it is the established vanilla ice-cream of classification metrics.

As you correctly assess maximising AUC-ROC does not guarantee that we will get the better combination of sensitivity and specificity or any other particular metric. (If you are interested in the combination of the two you might want to consider using F-score) In most applications defining a correct metric is half the battle. For a probabilistic classifier, I would suggest you look into using a proper scoring rule like the Brier score. Please note that accuracy is usually not a good metric because it is an improper rule (ie. maximising it does not guarantee we are as close as possible to the true probability distributions of the data).

Additional notes particular to the use case present:

1. For Support Vector Machines AUC-ROC is a bit fuzzy as a concept. SVMs natively do hard class-assignment and have to rely to isotonic regression or Platt scaling approaches to generate probability distributions over classes. This adds an additional layer of complexity.
2. At first instance, I could suggest using a "standard" regularisation approach like Elastic-net or Least-angle regression (to directly control for the $n \ll p$ scenario you outline. The techniques will do the "variable selection" internally through regularisation.
3. The sensitivity and specificity values you see in the output of rfe are related to a $0.5$ probability threshold. It is plausible you can get better performance using another threshold.
4. AUC-ROC is probably better than just using sensitivity and specificity. Both sensitivity and specificity are improper scoring rules that are hard to properly optimise for. For example, correctly increasing the probability of an item belonging to particular class does not increase neither sensitivity or specificity unless the increase passes an arbitrary threshold (usually $0.5$). This correct change in the estimated probability will be directly reflected on the AUC-ROC value.
5. ROC, ie. Receiver operating characteristic is used instead of simply AUC Area-Under-the-Curve because AUC can be easily defined for many other metrics too. A common one is AUC-PR, the Area-Under-the-Curve for the Precision Recall curve - a quite popular metric for imbalanced learning (eg. see Davis and Goadrich's The Relationship Between Precision-Recall and ROC Curves for more on this.)