1
$\begingroup$

I'm doing balanced (70%-30%) binary classification (yes/no) I'm trying to combine caret train objects with PRROC's pr.curve function. I'm using a confusion matrix to determine which class is labelled as "positive" to make sure I put the right class in the score.class0 and score.class1 arguments.

On my model, I have a high accuracy, recall and specificity as read in the confusion matrix. However, the result of the AUC from the prcurve function is extremely low.

here's a reprex of how my model is constructed, with more "predictable" results using naive bayes, but the AUC PRcurve results are much much lower for all the models I'm using (glm, knn, class.tree,...)

library(caret)
library(PRROC)

set.seed(73, sample.kind = "Rounding")

sonar_index <- createDataPartition(Sonar$Class, times = 1, p= 0.2, list = F)
sonar_train <- Sonar[-sonar_index,]
sonar_test <- Sonar[sonar_index,]

ctrl <- trainControl(method="cv", summaryFunction=twoClassSummary, classProbs=T,
                         savePredictions = T)

nb <- train(Class ~ V17 + V13,
                data = sonar_train, method = "nb", trControl = ctrl)

cm <- confusionMatrix(predict(nb, sonar_test), sonar_test$Class)

plot(pr.curve(scores.class0 = nb$pred$M[sonar_test$Class == "M"], 
              scores.class1 = nb$pred$R[sonar_test$Class == "R"],
              curve = T))
Improve this question
$\endgroup$
2
  • $\begingroup$ Note that the shape of a precision-recall curve can be more informative than the area under it. See this page and pages linked from there. Also, interpreting the AUPRC value isn't quite so straightforward as is interpreting the AUROC value. It might help to show both the PR curve and the corresponding ROC curve to provide more context. $\endgroup$
    – EdM
    Commented May 17, 2020 at 21:23
  • $\begingroup$ Thank for the link to the answer, it helps, although I also get a very low result for the AUC ROC curve. Which is weird because when I use pROC's auc() function on my model, results are higher. So I know I'm doing something wrong, I just don't know where $\endgroup$
    – bolleke
    Commented May 17, 2020 at 21:43

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.