I have a number of questions on the ROC curves when being used to evaluate a model.
My understanding of them is they can be used to determine the probability cutoff when classifying a row in a dataset to a class. For example if I had a dataset which was skewed 70%-30% in a class then when classifying without any investigation a good cutoff might be 70%. I am using the
ROCR library which can be found here at 1 and 2 for the ROC curve and the
yardstick library 3 to determine the AUC. A reproducible example is below this. I think, I understand the concept of ROC and its aims to provide a trade off between sensitivity and specificity. My questions are concerned with the implementation
My question is threefold
Given I get an AUC of 0.699 for a decision tree does this mean when predicting new examples that I should set the value of a class to positive when the probability is over 0.699. In my example below, all people who had a probability greater than 0.699 should be assigned to the class "Good" and everyone else assigned to the class "Bad"
If implementing cross validation, should the ROC curve be implemented on each train-test validation set and the average taken across those sets or will applying the technique to the final model be sufficient?
Finally I seem to be having problems with my ROC graph. As stated earlier the AUC for this is 0.699 which should suggest I get a concave graph. THe problem lies with the line
ROCR::prediction(assess_dat$Good, assess_dat$Class). If i switch it to
ROCR::prediction(assess_dat$Bad, assess_dat$Class)it seems to be OK. In that case is the first argument of this function the probability that a class does not belong to your class of interest and the second argument the Class you are focused on predicting?
library(caret) library(dplyr) library(yardstick) data(GermanCredit) options(stringsAsFactors = FALSE) table(GermanCredit$Class) #> #> Bad Good #> 300 700 levels(GermanCredit$Class) #>  "Bad" "Good" # I'm interested in the guys with good credit so i relevel the factor GermanCredit$Class <- relevel(GermanCredit$Class, ref = "Good") # create train and test set set.seed(3456) trainIndex <- createDataPartition(GermanCredit$Class, p = .8, list = FALSE, times = 1) train <- GermanCredit[ trainIndex,] test <- GermanCredit[-trainIndex,] # Train the model with 10 fold cross validation model <- train( Class ~., data = train, method = "rpart", trControl = trainControl("cv", number = 10) ) # Adds the model predictions and the model output to the dataframe assess_dat <- test %>% mutate(pred = predict(model, newdata = ., type = 'raw')) %>% bind_cols(predict(model, newdata = ., type = 'prob')) %>% select(Class, pred, Good, Bad) auc <- assess_dat %>% yardstick::roc_auc(truth = Class, Good) %>% select(auc = .estimate) # Have a look at the ROC curve # Probability of predictions vs the truth # In this case where we are trying to predict good pred <- ROCR::prediction(assess_dat$Good, assess_dat$Class) perf <- ROCR::performance(pred, "tpr", "fpr") perf_df <- data.frame(firstname.lastname@example.org, email@example.com) names(perf_df) <- c("tpr", "fpr") ggplot(perf_df, aes(x = fpr, y = tpr)) + geom_line() + geom_abline(intercept = 0, slope = 1, lty = 3) + xlab("False Positive Rate") + ylab("True Positive Rate") + ggtitle("ROC Curve auc = 0.699") + theme_minimal()
# Print Results confusionMatrix(assess_dat$pred, assess_dat$Class) #> Confusion Matrix and Statistics #> #> Reference #> Prediction Good Bad #> Good 128 49 #> Bad 12 11 #> #> Accuracy : 0.695 #> 95% CI : (0.6261, 0.758) #> No Information Rate : 0.7 #> P-Value [Acc > NIR] : 0.5953 #> #> Kappa : 0.1185 #> Mcnemar's Test P-Value : 4.04e-06 #> #> Sensitivity : 0.9143 #> Specificity : 0.1833 #> Pos Pred Value : 0.7232 #> Neg Pred Value : 0.4783 #> Prevalence : 0.7000 #> Detection Rate : 0.6400 #> Detection Prevalence : 0.8850 #> Balanced Accuracy : 0.5488 #> #> 'Positive' Class : Good #> auc #> # A tibble: 1 x 1 #> auc #> <dbl> #> 1 0.699
Created on 2018-10-26 by the reprex package (v0.2.1)