# Using An ROC Curve to Evaluate a model

I have a number of questions on the ROC curves when being used to evaluate a model. My understanding of them is they can be used to determine the probability cutoff when classifying a row in a dataset to a class. For example if I had a dataset which was skewed 70%-30% in a class then when classifying without any investigation a good cutoff might be 70%. I am using the ROCR library which can be found here at 1 and 2 for the ROC curve and the yardstick library 3 to determine the AUC. A reproducible example is below this. I think, I understand the concept of ROC and its aims to provide a trade off between sensitivity and specificity. My questions are concerned with the implementation

My question is threefold

• Given I get an AUC of 0.699 for a decision tree does this mean when predicting new examples that I should set the value of a class to positive when the probability is over 0.699. In my example below, all people who had a probability greater than 0.699 should be assigned to the class "Good" and everyone else assigned to the class "Bad"

• If implementing cross validation, should the ROC curve be implemented on each train-test validation set and the average taken across those sets or will applying the technique to the final model be sufficient?

• Finally I seem to be having problems with my ROC graph. As stated earlier the AUC for this is 0.699 which should suggest I get a concave graph. THe problem lies with the line ROCR::prediction(assess_dat$$Good, assess_dat$$Class). If i switch it to ROCR::prediction(assess_dat$$Bad, assess_dat$$Class) it seems to be OK. In that case is the first argument of this function the probability that a class does not belong to your class of interest and the second argument the Class you are focused on predicting?

library(caret)
library(dplyr)
library(yardstick)
data(GermanCredit)
options(stringsAsFactors = FALSE)

table(GermanCredit$$Class) #> #> Bad Good #> 300 700 levels(GermanCredit$$Class)

# I'm interested in the guys with good credit so i relevel the factor
GermanCredit$$Class <- relevel(GermanCredit$$Class, ref = "Good")

# create train and test set
set.seed(3456)
trainIndex <- createDataPartition(GermanCredit$Class, p = .8, list = FALSE, times = 1) train <- GermanCredit[ trainIndex,] test <- GermanCredit[-trainIndex,] # Train the model with 10 fold cross validation model <- train( Class ~., data = train, method = "rpart", trControl = trainControl("cv", number = 10) ) # Adds the model predictions and the model output to the dataframe assess_dat <- test %>% mutate(pred = predict(model, newdata = ., type = 'raw')) %>% bind_cols(predict(model, newdata = ., type = 'prob')) %>% select(Class, pred, Good, Bad) auc <- assess_dat %>% yardstick::roc_auc(truth = Class, Good) %>% select(auc = .estimate) # Have a look at the ROC curve # Probability of predictions vs the truth # In this case where we are trying to predict good pred <- ROCR::prediction(assess_dat$$Good, assess_dat$$Class) perf <- ROCR::performance(pred, "tpr", "fpr") perf_df <- data.frame([email protected], [email protected]) names(perf_df) <- c("tpr", "fpr") ggplot(perf_df, aes(x = fpr, y = tpr)) + geom_line() + geom_abline(intercept = 0, slope = 1, lty = 3) + xlab("False Positive Rate") + ylab("True Positive Rate") + ggtitle("ROC Curve auc = 0.699") + theme_minimal()  # Print Results confusionMatrix(assess_dat$$pred, assess_dat$$Class) #> Confusion Matrix and Statistics #> #> Reference #> Prediction Good Bad #> Good 128 49 #> Bad 12 11 #> #> Accuracy : 0.695 #> 95% CI : (0.6261, 0.758) #> No Information Rate : 0.7 #> P-Value [Acc > NIR] : 0.5953 #> #> Kappa : 0.1185 #> Mcnemar's Test P-Value : 4.04e-06 #> #> Sensitivity : 0.9143 #> Specificity : 0.1833 #> Pos Pred Value : 0.7232 #> Neg Pred Value : 0.4783 #> Prevalence : 0.7000 #> Detection Rate : 0.6400 #> Detection Prevalence : 0.8850 #> Balanced Accuracy : 0.5488 #> #> 'Positive' Class : Good #> auc #> # A tibble: 1 x 1 #> auc #> <dbl> #> 1 0.699  Created on 2018-10-26 by the reprex package (v0.2.1) • 0.699 is the AUROC (Area Under the Receiver Operating Characteristic Curve), what you are asking for is the threshold which should be somewhere in the output / object. You can see it in the plot though (which should be reversed), the optimal threshold is slightly smaller than 0.5. Commented Oct 26, 2018 at 8:12 ## 1 Answer problems with my ROC graph Try altering the labels to change the direction of the graph pred = prediction(assess_dat, assess_dat$Good, label.ordering = c("Good","Class")