2
$\begingroup$

I have a number of questions on the ROC curves when being used to evaluate a model. My understanding of them is they can be used to determine the probability cutoff when classifying a row in a dataset to a class. For example if I had a dataset which was skewed 70%-30% in a class then when classifying without any investigation a good cutoff might be 70%. I am using the ROCR library which can be found here at 1 and 2 for the ROC curve and the yardstick library 3 to determine the AUC. A reproducible example is below this. I think, I understand the concept of ROC and its aims to provide a trade off between sensitivity and specificity. My questions are concerned with the implementation

My question is threefold

  • Given I get an AUC of 0.699 for a decision tree does this mean when predicting new examples that I should set the value of a class to positive when the probability is over 0.699. In my example below, all people who had a probability greater than 0.699 should be assigned to the class "Good" and everyone else assigned to the class "Bad"

  • If implementing cross validation, should the ROC curve be implemented on each train-test validation set and the average taken across those sets or will applying the technique to the final model be sufficient?

  • Finally I seem to be having problems with my ROC graph. As stated earlier the AUC for this is 0.699 which should suggest I get a concave graph. THe problem lies with the line ROCR::prediction(assess_dat$Good, assess_dat$Class). If i switch it to ROCR::prediction(assess_dat$Bad, assess_dat$Class) it seems to be OK. In that case is the first argument of this function the probability that a class does not belong to your class of interest and the second argument the Class you are focused on predicting?

library(caret)
library(dplyr)
library(yardstick)
data(GermanCredit)
options(stringsAsFactors = FALSE)

table(GermanCredit$Class)
#> 
#>  Bad Good 
#>  300  700
levels(GermanCredit$Class)
#> [1] "Bad"  "Good"

# I'm interested in the guys with good credit so i relevel the factor
GermanCredit$Class <- relevel(GermanCredit$Class, ref = "Good")

# create train and test set
set.seed(3456)
trainIndex <- createDataPartition(GermanCredit$Class, p = .8, 
                                  list = FALSE, 
                                  times = 1)

train <- GermanCredit[ trainIndex,]
test  <- GermanCredit[-trainIndex,]

# Train the model with 10 fold cross validation
model <- train(
  Class ~., data = train,
  method = "rpart",
  trControl = trainControl("cv", number = 10)
)

# Adds the model predictions and the model output to the dataframe
assess_dat <- test %>%
  mutate(pred = predict(model, newdata = .,  type = 'raw')) %>%
  bind_cols(predict(model, newdata = .,  type = 'prob')) %>% 
  select(Class, pred, Good, Bad)

auc <- assess_dat %>% 
  yardstick::roc_auc(truth = Class, Good) %>% 
  select(auc = .estimate)

# Have a look at the ROC curve
# Probability of predictions vs the truth
# In this case where we are trying to predict good
pred <- ROCR::prediction(assess_dat$Good, assess_dat$Class)
perf <- ROCR::performance(pred, "tpr", "fpr")
perf_df <- data.frame([email protected], [email protected])
names(perf_df) <- c("tpr", "fpr")

ggplot(perf_df, aes(x = fpr, y = tpr)) +
  geom_line() +
  geom_abline(intercept = 0, slope = 1, lty = 3) +
  xlab("False Positive Rate") +
  ylab("True Positive Rate") +
  ggtitle("ROC Curve auc = 0.699") +
  theme_minimal()

# Print Results
confusionMatrix(assess_dat$pred, assess_dat$Class)
#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction Good Bad
#>       Good  128  49
#>       Bad    12  11
#>                                          
#>                Accuracy : 0.695          
#>                  95% CI : (0.6261, 0.758)
#>     No Information Rate : 0.7            
#>     P-Value [Acc > NIR] : 0.5953         
#>                                          
#>                   Kappa : 0.1185         
#>  Mcnemar's Test P-Value : 4.04e-06       
#>                                          
#>             Sensitivity : 0.9143         
#>             Specificity : 0.1833         
#>          Pos Pred Value : 0.7232         
#>          Neg Pred Value : 0.4783         
#>              Prevalence : 0.7000         
#>          Detection Rate : 0.6400         
#>    Detection Prevalence : 0.8850         
#>       Balanced Accuracy : 0.5488         
#>                                          
#>        'Positive' Class : Good           
#> 
auc
#> # A tibble: 1 x 1
#>     auc
#>   <dbl>
#> 1 0.699

Created on 2018-10-26 by the reprex package (v0.2.1)

$\endgroup$
1
  • 1
    $\begingroup$ 0.699 is the AUROC (Area Under the Receiver Operating Characteristic Curve), what you are asking for is the threshold which should be somewhere in the output / object. You can see it in the plot though (which should be reversed), the optimal threshold is slightly smaller than 0.5. $\endgroup$ Oct 26, 2018 at 8:12

1 Answer 1

2
$\begingroup$

problems with my ROC graph

Try altering the labels to change the direction of the graph

pred = prediction(assess_dat, assess_dat$Good, label.ordering = c("Good","Class")
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.