1
$\begingroup$

I have somewhat of an odd question.

I'm currently working on a dataset and after training my model, I'm trying to get the correct AUC and I'm using the RocR package to get that value.

Im using these commands

  pred <- predict(model, testData,type='prob')  
  pr=prediction(pred[,2],testData$var)  
  prf <- performance(pr, measure = "tpr", x.measure = "fpr")  

  auc <- performance(pr, measure = "auc")  
  auc <- auc@y.values[[1]]  

I've noticed that whether I add the type='prob' changes the value of my auc, meaning that if i predict just the binary output instead of the probabilities, the AUC changes. So my question is which value should I use? I'm a bit lost.

$\endgroup$
1
$\begingroup$

What's your model? Note that predict is not a ROCR function, ROCR starts from prediction. It's your job to specify the model correctly. Of course, what the model generates have a significant impact on you AUC statistics, because you're comparing your prediction against testData$var. You'll need to think what exactly you want to predict. ROC is just a statistical tool, it's garbage in and garbage out, so make sure don't give garbage inputs.

What's testData$var? It's supposed to be the classified labels. Are they related to your binary or probability prediction (I don't know what model you're using)?

Note that you don't choose your AUC statistics, you choose your model that maximize AUC statistics as much as possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.