I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this using
performance, but first you need to create an ROC curve. However, every time I do this my AUROC=1. I tried both ways and get the same results. The model should have poor performance as seen by the confusion matrix so what does this AUC mean?
Here is a reproducible example with poor performance but great AUC.
library(dplyr) library(caret) library(pROC) library(ROCR) attach(attitude) #create class imbalance df<- attitude %>% mutate(outcome=ifelse(between(rating, 62,67),"case","control")) #rf needs outcome as a factor df$outcome <- as.factor(df$outcome) set.seed(3949) #create train set df_train <- sample_frac(df, 0.7) #create test set idx <- as.numeric(rownames(df_train)) df_test <- df[-idx, ] #set up trControl ctrl <- trainControl(method = "cv", number = 5, savePredictions = TRUE, summaryFunction = twoClassSummary, classProbs = TRUE) #create tuned model set.seed(3949) model <- train(outcome ~ ., data=df_train, method= "rf", trControl = ctrl, preProc=c("center","scale"), metric="ROC", tuneGrid = data.frame(mtry = 2), importance=TRUE) confusionMatrix(model$finalModel$predicted, df_train$outcome, positive="case") #not great performance
r<-roc(model$finalModel$predicted, model$finalModel$votes[,2], levels=c("control","case"), direction=">") #AUC=1
#the train AUC rf_p_train <- predict(model, type="prob")[,2] rf_pr_train <- prediction(rf_p_train, df_train$outcome) r_auc_train <- performance(rf_pr_train, measure = "auc")@y.values[] r_auc_train #AUC=1
I'm just suspicious because when I pull out the sensitivities and specificities at all the thresholds, at any given threshold, sensitivity=1.0 or specificity <1.0 and vice versa. And this just seems wrong...? Is it my code incorrect? *Note: I want to look at the full training data and not the cross validation results.
List of thresholds against sensitivity and specificity demonstrating sensitivity or specificity always =1.0
coordinates <- coords(r, x = "all", input = "threshold", ret = c("threshold", "sen","spe")) thresholds <- as.data.frame(t(coordinates))