2
$\begingroup$

I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this using roc or performance, but first you need to create an ROC curve. However, every time I do this my AUROC=1. I tried both ways and get the same results. The model should have poor performance as seen by the confusion matrix so what does this AUC mean?

Here is a reproducible example with poor performance but great AUC.

library(dplyr)
library(caret)
library(pROC)
library(ROCR)

attach(attitude)
#create class imbalance
df<- attitude %>% mutate(outcome=ifelse(between(rating, 
62,67),"case","control"))
#rf needs outcome as a factor
df$outcome <- as.factor(df$outcome)

set.seed(3949)
#create train set
df_train <- sample_frac(df, 0.7)
#create test set
idx <- as.numeric(rownames(df_train))
df_test <- df[-idx, ]

#set up trControl
ctrl <- trainControl(method = "cv",
                 number = 5,
                 savePredictions = TRUE,
                 summaryFunction = twoClassSummary,
                 classProbs = TRUE)

#create tuned model
set.seed(3949)
model <- train(outcome ~ ., data=df_train, 
                    method= "rf",
                    trControl = ctrl, 
                    preProc=c("center","scale"), 
                    metric="ROC",
                    tuneGrid = data.frame(mtry = 2),
                    importance=TRUE)
confusionMatrix(model$finalModel$predicted, df_train$outcome, positive="case") #not great performance

Method 1:

r<-roc(model$finalModel$predicted, model$finalModel$votes[,2],
   levels=c("control","case"), direction=">") #AUC=1

Method 2:

#the train AUC
rf_p_train <- predict(model, type="prob")[,2]
rf_pr_train <- prediction(rf_p_train, df_train$outcome)
r_auc_train <- performance(rf_pr_train, measure = "auc")@y.values[[1]] 
r_auc_train   #AUC=1

I'm just suspicious because when I pull out the sensitivities and specificities at all the thresholds, at any given threshold, sensitivity=1.0 or specificity <1.0 and vice versa. And this just seems wrong...? Is it my code incorrect? *Note: I want to look at the full training data and not the cross validation results.

List of thresholds against sensitivity and specificity demonstrating sensitivity or specificity always =1.0

coordinates <- coords(r, x = "all", input = "threshold", ret = c("threshold", "sen","spe"))
thresholds <- as.data.frame(t(coordinates))
$\endgroup$
8
  • $\begingroup$ On the full training data, you will have to consult out-of-bag AUC. Usual insample performance of a random forest is indeed too good to be true. A bit like if you would do 3-nearest-neighbour classification including the observation itself. $\endgroup$
    – Michael M
    Jun 17, 2020 at 19:24
  • $\begingroup$ So to understand the AUC, I have to look at OOB. But if I'm using the ROC as a means to an end (i.e. to look at my thresholds) is that ok? $\endgroup$
    – PleaseHelp
    Jun 17, 2020 at 19:28
  • 1
    $\begingroup$ I'd say so, although I don't know much about your specific situation. Important for honest estimates of performance is that there are no dependencies across rows (clusters etc.). $\endgroup$
    – Michael M
    Jun 17, 2020 at 19:59
  • 1
    $\begingroup$ Without the data your example is not reproducible... and it's going to be tough to answer properly. $\endgroup$
    – Calimo
    Jun 18, 2020 at 6:39
  • 1
    $\begingroup$ I found the twoClassSummary with a lowercase t in caret. I assume it was just a typo. $\endgroup$
    – Calimo
    Jun 19, 2020 at 9:59

1 Answer 1

2
$\begingroup$

There are several aspects to unravel here.

Building the ROC curve correctly (Method 1)

In order to build the ROC curve, you need to provide two vectors: the ground truth, and a numeric estimate of your predictions. Looking at your Method 1 code:

r <- roc(model$finalModel$predicted, model$finalModel$votes[,2],
         levels=c("control", "case"), direction=">")

You are supplying both the predicted class model$finalModel$predicted and the numeric estimates (here votes with probability to be a control, model$finalModel$votes[,2]).

Of course these are going to be in perfect agreement, giving you a meaningless AUC of 1.0.

With Method 2 and the confusionMatrix you are correctly using the ground truth (df_train$outcome) so that's good.

Choosing the right threshold

The confusionMatrix function gives you the following performance:

        Sensitivity : 0.16667         
        Specificity : 1.00000  

However ROC analysis gives you one advantage: you can choose a better threshold!

With pROC's coords function you can see the following:

> r <- roc(df_train$outcome, rf_p_train,
+          levels=c("control", "case"), direction=">")
> coords(r)
   threshold specificity sensitivity
1        Inf  0.00000000   1.0000000
2      0.958  0.06666667   1.0000000
3      0.946  0.13333333   1.0000000
4      0.932  0.20000000   1.0000000
5      0.920  0.33333333   1.0000000
6      0.915  0.40000000   1.0000000
7      0.910  0.46666667   1.0000000
8      0.894  0.53333333   1.0000000
9      0.878  0.66666667   1.0000000
10     0.872  0.73333333   1.0000000
11     0.863  0.80000000   1.0000000
12     0.853  0.86666667   1.0000000
13     0.839  0.93333333   1.0000000
14     0.558  1.00000000   1.0000000
15     0.262  1.00000000   0.8333333
16     0.235  1.00000000   0.6666667
17     0.233  1.00000000   0.5000000
18     0.219  1.00000000   0.3333333
19     0.176  1.00000000   0.1666667
20      -Inf  1.00000000   0.0000000

You can see the threshold 0.176 matches the performance of the contingency table with a sensitivity of 0.16. However there is a better threshold:

14     0.558  1.00000000   1.0000000

I don't know why caret chose a threshold around 0.176 instead for the classification, but for sure with a threshold of 0.5, you get a perfect classification!

Resubstitution

You seem to be aware of it as you state "I want to look at the full training data and not the cross validation results", however for the sake of completeness of this answer: using the training data to estimate the performance of a model, also called a resubstitution estimate, will give you an overoptimistic result. So yes, definitely, your AUC is too good to be true.

Getting the correct AUC

You already defined the test set in df_test. Let's get the model predictions on that dataset:

rf_p_test <- predict(model, type="prob", newdata = df_test)[,1]

I used column 1 which contains the predictions to be a 'case', which seems more natural to me. Now we can build the ROC curve:

r <- roc(df_test$outcome, rf_p_test)

And get the AUC:

> auc(r)
# Area under the curve: 1

So it turns out the randomForest model was pretty good at capturing the essence of the problem.

$\endgroup$
3
  • $\begingroup$ My apologies for the delay. Thank you so much for taking the time to walk me through this example!! Just to clarify, in your roc() function you used rf_p_train which is generated from predict(); how is different from model$finalModel$votes[,2]? I've seen both methods used synonymously, but the latter yields difference performances for different thresholds. $\endgroup$
    – PleaseHelp
    Jun 23, 2020 at 20:57
  • $\begingroup$ *should be model$finalModel$votes[,1] $\endgroup$
    – PleaseHelp
    Jun 23, 2020 at 21:15
  • 1
    $\begingroup$ I don't understand the difference either. Probably worth asking as a caret-specific question. But in any case, predict is the standard, idiomatic way to do it in R and you can go for it safely. $\endgroup$
    – Calimo
    Jun 24, 2020 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.