Is my AUC too good to be true?

Question

I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this using roc or performance, but first you need to create an ROC curve. However, every time I do this my AUROC=1. I tried both ways and get the same results. The model should have poor performance as seen by the confusion matrix so what does this AUC mean?

Here is a reproducible example with poor performance but great AUC.

library(dplyr)
library(caret)
library(pROC)
library(ROCR)

attach(attitude)
#create class imbalance
df<- attitude %>% mutate(outcome=ifelse(between(rating, 
62,67),"case","control"))
#rf needs outcome as a factor
df$outcome <- as.factor(df$outcome)

set.seed(3949)
#create train set
df_train <- sample_frac(df, 0.7)
#create test set
idx <- as.numeric(rownames(df_train))
df_test <- df[-idx, ]

#set up trControl
ctrl <- trainControl(method = "cv",
                 number = 5,
                 savePredictions = TRUE,
                 summaryFunction = twoClassSummary,
                 classProbs = TRUE)

#create tuned model
set.seed(3949)
model <- train(outcome ~ ., data=df_train, 
                    method= "rf",
                    trControl = ctrl, 
                    preProc=c("center","scale"), 
                    metric="ROC",
                    tuneGrid = data.frame(mtry = 2),
                    importance=TRUE)
confusionMatrix(model$finalModel$predicted, df_train$outcome, positive="case") #not great performance

Method 1:

r<-roc(model$finalModel$predicted, model$finalModel$votes[,2],
   levels=c("control","case"), direction=">") #AUC=1

Method 2:

#the train AUC
rf_p_train <- predict(model, type="prob")[,2]
rf_pr_train <- prediction(rf_p_train, df_train$outcome)
r_auc_train <- performance(rf_pr_train, measure = "auc")@y.values[[1]] 
r_auc_train   #AUC=1

I'm just suspicious because when I pull out the sensitivities and specificities at all the thresholds, at any given threshold, sensitivity=1.0 or specificity <1.0 and vice versa. And this just seems wrong...? Is it my code incorrect? *Note: I want to look at the full training data and not the cross validation results.

List of thresholds against sensitivity and specificity demonstrating sensitivity or specificity always =1.0

coordinates <- coords(r, x = "all", input = "threshold", ret = c("threshold", "sen","spe"))
thresholds <- as.data.frame(t(coordinates))

Answer 1

There are several aspects to unravel here.

Building the ROC curve correctly (Method 1)

In order to build the ROC curve, you need to provide two vectors: the ground truth, and a numeric estimate of your predictions. Looking at your Method 1 code:

r <- roc(model$finalModel$predicted, model$finalModel$votes[,2],
         levels=c("control", "case"), direction=">")

You are supplying both the predicted class model$finalModel$predicted and the numeric estimates (here votes with probability to be a control, model$finalModel$votes[,2]).

Of course these are going to be in perfect agreement, giving you a meaningless AUC of 1.0.

With Method 2 and the confusionMatrix you are correctly using the ground truth (df_train$outcome) so that's good.

Choosing the right threshold

The confusionMatrix function gives you the following performance:

        Sensitivity : 0.16667         
        Specificity : 1.00000  

However ROC analysis gives you one advantage: you can choose a better threshold!

With pROC's coords function you can see the following:

> r <- roc(df_train$outcome, rf_p_train,
+          levels=c("control", "case"), direction=">")
> coords(r)
   threshold specificity sensitivity
1        Inf  0.00000000   1.0000000
2      0.958  0.06666667   1.0000000
3      0.946  0.13333333   1.0000000
4      0.932  0.20000000   1.0000000
5      0.920  0.33333333   1.0000000
6      0.915  0.40000000   1.0000000
7      0.910  0.46666667   1.0000000
8      0.894  0.53333333   1.0000000
9      0.878  0.66666667   1.0000000
10     0.872  0.73333333   1.0000000
11     0.863  0.80000000   1.0000000
12     0.853  0.86666667   1.0000000
13     0.839  0.93333333   1.0000000
14     0.558  1.00000000   1.0000000
15     0.262  1.00000000   0.8333333
16     0.235  1.00000000   0.6666667
17     0.233  1.00000000   0.5000000
18     0.219  1.00000000   0.3333333
19     0.176  1.00000000   0.1666667
20      -Inf  1.00000000   0.0000000

You can see the threshold 0.176 matches the performance of the contingency table with a sensitivity of 0.16. However there is a better threshold:

14     0.558  1.00000000   1.0000000

I don't know why caret chose a threshold around 0.176 instead for the classification, but for sure with a threshold of 0.5, you get a perfect classification!

Resubstitution

You seem to be aware of it as you state "I want to look at the full training data and not the cross validation results", however for the sake of completeness of this answer: using the training data to estimate the performance of a model, also called a resubstitution estimate, will give you an overoptimistic result. So yes, definitely, your AUC is too good to be true.

Getting the correct AUC

You already defined the test set in df_test. Let's get the model predictions on that dataset:

rf_p_test <- predict(model, type="prob", newdata = df_test)[,1]

I used column 1 which contains the predictions to be a 'case', which seems more natural to me. Now we can build the ROC curve:

r <- roc(df_test$outcome, rf_p_test)

And get the AUC:

> auc(r)
# Area under the curve: 1

So it turns out the randomForest model was pretty good at capturing the essence of the problem.