0
$\begingroup$

This question already has an answer here:

In the R package ModelMetrics, the auc score as shown in the documentation takes only two inputs;

aucScore <- auc(actual=actuallabels, predicted=predictedlabels)

where the inputs are pretty self explanatory. However, how is the AUC score even calculated here? From my understanding, do we not need the class "probabilities" (scores) produced from the model in order to graph the ROC curve in the first place? Each point on the ROC curve is the TPR vs. the FPR, at varying thresholds of class "probabilities" to trigger a "positive" prediction, if my understanding is correct. So how can we chart the ROC curve and find the area under said curve if we don't have the class probabilities to derive each point on the ROC curve itself? Based off labels alone, I would find that it would be impossible to find this unless there's some underlying estimation going on.

I've also noticed that many other packages also compute the area under the ROC curve by taking only two vectors of labels with seemingly no class probabilities.

Thanks.

$\endgroup$

marked as duplicate by Sycorax, kjetil b halvorsen, Peter Flom Apr 10 '18 at 21:07

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ In the documentation for ModelMetrics, they use the probabilities: data(testDF); glmModel <- glm(y ~ ., data = testDF, family="binomial"); Preds <- predict(glmModel, type = 'response'); auc(testDF$y, Preds); $\endgroup$ – Matthew Drury Apr 10 '18 at 18:38
  • $\begingroup$ I see that in the documentation for glm models the AUROC score is indeed a probability. I guess my question would then become, why is it that I put in two vectors of labels only, and still get some sort of score? Perhaps the methodologies change based on input? This might be a different question, or at least, require an example. Sorry for that and the possible duplicate. $\endgroup$ – aranglol Apr 10 '18 at 18:53
  • $\begingroup$ Probably just because the code for computing the AUC type checks. As in, when you pass a vector of labels (zero and ones I assume) the code still runs, and gives you an answer. The zeros and ones can, after all, be interpreted as scores, so you really have a ROC curve with three points on it (two of those points are (0, 0) and (1, 1), so there is one non trivial point). $\endgroup$ – Matthew Drury Apr 10 '18 at 20:20
  • $\begingroup$ Ah, okay. So if I pass a vector of a factor variable, say "W" and "L" for win/loss respectively, then all that the code is doing is converting these factors to 0/1 respectively (or some other arbitrary coding scheme). I see now. Thanks for the clarification Matthew. $\endgroup$ – aranglol Apr 10 '18 at 21:32
0
$\begingroup$

Each point on the ROC curve is the TPR vs. the FPR, at varying thresholds of class "probabilities"

ROC analysis is non-parametric has no requirement for the classifier scores to be probabilities. It can accommodate any continuous or discrete scores, as long as the threshold that is used to make the actual decision can be varied.

Consequently, transformations that keep the ordering of the data points unchanged (such as linearly converting arbitrary scores to probabilities or vice-versa) do not alter the ROC curve.

The only "requirement" is to be able to vary the detection threshold.

$\endgroup$
  • $\begingroup$ What part of the quotation do you object to? Your reply does not appear to conflict with what OP has written. For example, you object that "ROC analysis makes no assumptions about the distribution of the classifier output" but none of those words appear on the quotation. Nor does OP mention "requirements." $\endgroup$ – Sycorax Apr 10 '18 at 18:45
  • $\begingroup$ Can you please elaborate on this? I still don't see how in any way how there is a threshold in the first place, if I input two vectors with labels of "Win" and "Loss" for example, one vector represents the actuals, the other the model predictions, how am I able to vary the threshold of what will eventually be labelled as "Win" and the other "Loss"? I just read the linked post from Sycorax, the post by Alexey Grigorev seems to clear up my confusion. It appears that from his explanation we do indeed need class "scores", if I am reading his post correctly. $\endgroup$ – aranglol Apr 10 '18 at 18:49
  • $\begingroup$ @Sycorax rephrased a bit, but basically I don't know where the OP got the idea that ROC needs probabilities in the first place. $\endgroup$ – Calimo Apr 10 '18 at 19:42
  • $\begingroup$ @aranglol I hope it's clearer now. If you have Win and Loss you can't vary the threshold so you're out of luck. But if you have "Loss", "Null", "Win" and "Jackpot", then you're good to go. $\endgroup$ – Calimo Apr 10 '18 at 19:45
  • $\begingroup$ I see what you mean now. I suppose when I stated: "From my understanding, do we not need the class "probabilities" (scores) produced from the model in order to graph the ROC curve in the first place" I put probabilities in quotations along with (scores) to note any sort of ranking systems (and not necessarily probabilities). I can see where I wasn't clear, sorry. The detection threshold being variable is very helpful though. The "probability" scores as reported by my model are necessary as this allows the variation in the threshold. Thanks Calimo. $\endgroup$ – aranglol Apr 10 '18 at 21:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.