Using An ROC Curve to Evaluate a model

I have a number of questions on the ROC curves when being used to evaluate a model. My understanding of them is they can be used to determine the probability cutoff when classifying a row in a dataset to a class. For example if I had a dataset which was skewed 70%-30% in a class then when classifying without any investigation a good cutoff might be 70%. I am using the ROCR library which can be found here at 1 and 2 for the ROC curve and the yardstick library 3 to determine the AUC. A reproducible example is below this. I think, I understand the concept of ROC and its aims to provide a trade off between sensitivity and specificity. My questions are concerned with the implementation

My question is threefold

• Given I get an AUC of 0.699 for a decision tree does this mean when predicting new examples that I should set the value of a class to positive when the probability is over 0.699. In my example below, all people who had a probability greater than 0.699 should be assigned to the class "Good" and everyone else assigned to the class "Bad"

• If implementing cross validation, should the ROC curve be implemented on each train-test validation set and the average taken across those sets or will applying the technique to the final model be sufficient?

• Finally I seem to be having problems with my ROC graph. As stated earlier the AUC for this is 0.699 which should suggest I get a concave graph. THe problem lies with the line ROCR::prediction(assess_dat$$Good, assess_dat$$Class). If i switch it to ROCR::prediction(assess_dat$$Bad, assess_dat$$Class) it seems to be OK. In that case is the first argument of this function the probability that a class does not belong to your class of interest and the second argument the Class you are focused on predicting?

library(caret)
library(dplyr)
library(yardstick)
data(GermanCredit)
options(stringsAsFactors = FALSE)

table(GermanCredit$$Class) #> #> Bad Good #> 300 700 levels(GermanCredit$$Class)
#>  "Bad"  "Good"

# I'm interested in the guys with good credit so i relevel the factor
GermanCredit$$Class <- relevel(GermanCredit$$Class, ref = "Good")

# create train and test set
set.seed(3456)
trainIndex <- createDataPartition(GermanCredit\$Class, p = .8,
list = FALSE,
times = 1)

train <- GermanCredit[ trainIndex,]
test  <- GermanCredit[-trainIndex,]

# Train the model with 10 fold cross validation
model <- train(
Class ~., data = train,
method = "rpart",
trControl = trainControl("cv", number = 10)
)

# Adds the model predictions and the model output to the dataframe
assess_dat <- test %>%
mutate(pred = predict(model, newdata = .,  type = 'raw')) %>%
bind_cols(predict(model, newdata = .,  type = 'prob')) %>%
select(Class, pred, Good, Bad)

auc <- assess_dat %>%
yardstick::roc_auc(truth = Class, Good) %>%
select(auc = .estimate)

# Have a look at the ROC curve
# Probability of predictions vs the truth
# In this case where we are trying to predict good
pred <- ROCR::prediction(assess_dat$$Good, assess_dat$$Class)
perf <- ROCR::performance(pred, "tpr", "fpr")
perf_df <- data.frame(perf@x.values, perf@y.values)
names(perf_df) <- c("tpr", "fpr")

ggplot(perf_df, aes(x = fpr, y = tpr)) +
geom_line() +
geom_abline(intercept = 0, slope = 1, lty = 3) +
xlab("False Positive Rate") +
ylab("True Positive Rate") +
ggtitle("ROC Curve auc = 0.699") +
theme_minimal() # Print Results
confusionMatrix(assess_dat$$pred, assess_dat$$Class)
#> Confusion Matrix and Statistics
#>
#>           Reference
#> Prediction Good Bad
#>       Good  128  49
#>       Bad    12  11
#>
#>                Accuracy : 0.695
#>                  95% CI : (0.6261, 0.758)
#>     No Information Rate : 0.7
#>     P-Value [Acc > NIR] : 0.5953
#>
#>                   Kappa : 0.1185
#>  Mcnemar's Test P-Value : 4.04e-06
#>
#>             Sensitivity : 0.9143
#>             Specificity : 0.1833
#>          Pos Pred Value : 0.7232
#>          Neg Pred Value : 0.4783
#>              Prevalence : 0.7000
#>          Detection Rate : 0.6400
#>    Detection Prevalence : 0.8850
#>       Balanced Accuracy : 0.5488
#>
#>        'Positive' Class : Good
#>
auc
#> # A tibble: 1 x 1
#>     auc
#>   <dbl>
#> 1 0.699

Created on 2018-10-26 by the reprex package (v0.2.1)

• 0.699 is the AUROC (Area Under the Receiver Operating Characteristic Curve), what you are asking for is the threshold which should be somewhere in the output / object. You can see it in the plot though (which should be reversed), the optimal threshold is slightly smaller than 0.5. – user2974951 Oct 26 '18 at 8:12