I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this using roc
or performance
, but first you need to create an ROC curve. However, every time I do this my AUROC=1. I tried both ways and get the same results. The model should have poor performance as seen by the confusion matrix so what does this AUC mean?
Here is a reproducible example with poor performance but great AUC.
library(dplyr)
library(caret)
library(pROC)
library(ROCR)
attach(attitude)
#create class imbalance
df<- attitude %>% mutate(outcome=ifelse(between(rating,
62,67),"case","control"))
#rf needs outcome as a factor
df$outcome <- as.factor(df$outcome)
set.seed(3949)
#create train set
df_train <- sample_frac(df, 0.7)
#create test set
idx <- as.numeric(rownames(df_train))
df_test <- df[-idx, ]
#set up trControl
ctrl <- trainControl(method = "cv",
number = 5,
savePredictions = TRUE,
summaryFunction = twoClassSummary,
classProbs = TRUE)
#create tuned model
set.seed(3949)
model <- train(outcome ~ ., data=df_train,
method= "rf",
trControl = ctrl,
preProc=c("center","scale"),
metric="ROC",
tuneGrid = data.frame(mtry = 2),
importance=TRUE)
confusionMatrix(model$finalModel$predicted, df_train$outcome, positive="case") #not great performance
Method 1:
r<-roc(model$finalModel$predicted, model$finalModel$votes[,2],
levels=c("control","case"), direction=">") #AUC=1
#the train AUC
rf_p_train <- predict(model, type="prob")[,2]
rf_pr_train <- prediction(rf_p_train, df_train$outcome)
r_auc_train <- performance(rf_pr_train, measure = "auc")@y.values[[1]]
r_auc_train #AUC=1
I'm just suspicious because when I pull out the sensitivities and specificities at all the thresholds, at any given threshold, sensitivity=1.0 or specificity <1.0 and vice versa. And this just seems wrong...? Is it my code incorrect? *Note: I want to look at the full training data and not the cross validation results.
List of thresholds against sensitivity and specificity demonstrating sensitivity or specificity always =1.0
coordinates <- coords(r, x = "all", input = "threshold", ret = c("threshold", "sen","spe"))
thresholds <- as.data.frame(t(coordinates))
twoClassSummary
with a lowercaset
in caret. I assume it was just a typo. $\endgroup$