Using An ROC Curve to Evaluate a model

Question

I have a number of questions on the ROC curves when being used to evaluate a model. My understanding of them is they can be used to determine the probability cutoff when classifying a row in a dataset to a class. For example if I had a dataset which was skewed 70%-30% in a class then when classifying without any investigation a good cutoff might be 70%. I am using the ROCR library which can be found here at 1 and 2 for the ROC curve and the yardstick library 3 to determine the AUC. A reproducible example is below this. I think, I understand the concept of ROC and its aims to provide a trade off between sensitivity and specificity. My questions are concerned with the implementation

My question is threefold

• Given I get an AUC of 0.699 for a decision tree does this mean when predicting new examples that I should set the value of a class to positive when the probability is over 0.699. In my example below, all people who had a probability greater than 0.699 should be assigned to the class "Good" and everyone else assigned to the class "Bad"

• If implementing cross validation, should the ROC curve be implemented on each train-test validation set and the average taken across those sets or will applying the technique to the final model be sufficient?

• Finally I seem to be having problems with my ROC graph. As stated earlier the AUC for this is 0.699 which should suggest I get a concave graph. THe problem lies with the line ROCR::prediction(assess_dat$Good, assess_dat$Class). If i switch it to ROCR::prediction(assess_dat$Bad, assess_dat$Class) it seems to be OK. In that case is the first argument of this function the probability that a class does not belong to your class of interest and the second argument the Class you are focused on predicting?

library(caret)
library(dplyr)
library(yardstick)
data(GermanCredit)
options(stringsAsFactors = FALSE)

table(GermanCredit$Class)
#> 
#>  Bad Good 
#>  300  700
levels(GermanCredit$Class)
#> [1] "Bad"  "Good"

# I'm interested in the guys with good credit so i relevel the factor
GermanCredit$Class <- relevel(GermanCredit$Class, ref = "Good")

# create train and test set
set.seed(3456)
trainIndex <- createDataPartition(GermanCredit$Class, p = .8, 
                                  list = FALSE, 
                                  times = 1)

train <- GermanCredit[ trainIndex,]
test  <- GermanCredit[-trainIndex,]

# Train the model with 10 fold cross validation
model <- train(
  Class ~., data = train,
  method = "rpart",
  trControl = trainControl("cv", number = 10)
)

# Adds the model predictions and the model output to the dataframe
assess_dat <- test %>%
  mutate(pred = predict(model, newdata = .,  type = 'raw')) %>%
  bind_cols(predict(model, newdata = .,  type = 'prob')) %>% 
  select(Class, pred, Good, Bad)

auc <- assess_dat %>% 
  yardstick::roc_auc(truth = Class, Good) %>% 
  select(auc = .estimate)

# Have a look at the ROC curve
# Probability of predictions vs the truth
# In this case where we are trying to predict good
pred <- ROCR::prediction(assess_dat$Good, assess_dat$Class)
perf <- ROCR::performance(pred, "tpr", "fpr")
perf_df <- data.frame([email protected], [email protected])
names(perf_df) <- c("tpr", "fpr")

ggplot(perf_df, aes(x = fpr, y = tpr)) +
  geom_line() +
  geom_abline(intercept = 0, slope = 1, lty = 3) +
  xlab("False Positive Rate") +
  ylab("True Positive Rate") +
  ggtitle("ROC Curve auc = 0.699") +
  theme_minimal()

# Print Results
confusionMatrix(assess_dat$pred, assess_dat$Class)
#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction Good Bad
#>       Good  128  49
#>       Bad    12  11
#>                                          
#>                Accuracy : 0.695          
#>                  95% CI : (0.6261, 0.758)
#>     No Information Rate : 0.7            
#>     P-Value [Acc > NIR] : 0.5953         
#>                                          
#>                   Kappa : 0.1185         
#>  Mcnemar's Test P-Value : 4.04e-06       
#>                                          
#>             Sensitivity : 0.9143         
#>             Specificity : 0.1833         
#>          Pos Pred Value : 0.7232         
#>          Neg Pred Value : 0.4783         
#>              Prevalence : 0.7000         
#>          Detection Rate : 0.6400         
#>    Detection Prevalence : 0.8850         
#>       Balanced Accuracy : 0.5488         
#>                                          
#>        'Positive' Class : Good           
#> 
auc
#> # A tibble: 1 x 1
#>     auc
#>   <dbl>
#> 1 0.699

^{Created on 2018-10-26 by the reprex package (v0.2.1)}

Answer 1

problems with my ROC graph

Try altering the labels to change the direction of the graph

pred = prediction(assess_dat, assess_dat$Good, label.ordering = c("Good","Class")