0
$\begingroup$

In the R package ModelMetrics, the auc score as shown in the documentation takes only two inputs;

aucScore <- auc(actual=actuallabels, predicted=predictedlabels)

where the inputs are pretty self explanatory. However, how is the AUC score even calculated here? From my understanding, do we not need the class "probabilities" (scores) produced from the model in order to graph the ROC curve in the first place? Each point on the ROC curve is the TPR vs. the FPR, at varying thresholds of class "probabilities" to trigger a "positive" prediction, if my understanding is correct. So how can we chart the ROC curve and find the area under said curve if we don't have the class probabilities to derive each point on the ROC curve itself? Based off labels alone, I would find that it would be impossible to find this unless there's some underlying estimation going on.

I've also noticed that many other packages also compute the area under the ROC curve by taking only two vectors of labels with seemingly no class probabilities.

Thanks.

Improve this question
$\endgroup$
4
  • $\begingroup$ In the documentation for ModelMetrics, they use the probabilities: data(testDF); glmModel <- glm(y ~ ., data = testDF, family="binomial"); Preds <- predict(glmModel, type = 'response'); auc(testDF$y, Preds); $\endgroup$
    – Matthew Drury
    Commented Apr 10, 2018 at 18:38
  • $\begingroup$ I see that in the documentation for glm models the AUROC score is indeed a probability. I guess my question would then become, why is it that I put in two vectors of labels only, and still get some sort of score? Perhaps the methodologies change based on input? This might be a different question, or at least, require an example. Sorry for that and the possible duplicate. $\endgroup$
    – aranglol
    Commented Apr 10, 2018 at 18:53
  • $\begingroup$ Probably just because the code for computing the AUC type checks. As in, when you pass a vector of labels (zero and ones I assume) the code still runs, and gives you an answer. The zeros and ones can, after all, be interpreted as scores, so you really have a ROC curve with three points on it (two of those points are (0, 0) and (1, 1), so there is one non trivial point). $\endgroup$
    – Matthew Drury
    Commented Apr 10, 2018 at 20:20
  • $\begingroup$ Ah, okay. So if I pass a vector of a factor variable, say "W" and "L" for win/loss respectively, then all that the code is doing is converting these factors to 0/1 respectively (or some other arbitrary coding scheme). I see now. Thanks for the clarification Matthew. $\endgroup$
    – aranglol
    Commented Apr 10, 2018 at 21:32

1 Answer 1

Reset to default
0
$\begingroup$

Each point on the ROC curve is the TPR vs. the FPR, at varying thresholds of class "probabilities"

ROC analysis is non-parametric has no requirement for the classifier scores to be probabilities. It can accommodate any continuous or discrete scores, as long as the threshold that is used to make the actual decision can be varied.

Consequently, transformations that keep the ordering of the data points unchanged (such as linearly converting arbitrary scores to probabilities or vice-versa) do not alter the ROC curve.

The only "requirement" is to be able to vary the detection threshold.

Improve this answer
$\endgroup$
5
  • $\begingroup$ What part of the quotation do you object to? Your reply does not appear to conflict with what OP has written. For example, you object that "ROC analysis makes no assumptions about the distribution of the classifier output" but none of those words appear on the quotation. Nor does OP mention "requirements." $\endgroup$
    – Sycorax ♦
    Commented Apr 10, 2018 at 18:45
  • $\begingroup$ Can you please elaborate on this? I still don't see how in any way how there is a threshold in the first place, if I input two vectors with labels of "Win" and "Loss" for example, one vector represents the actuals, the other the model predictions, how am I able to vary the threshold of what will eventually be labelled as "Win" and the other "Loss"? I just read the linked post from Sycorax, the post by Alexey Grigorev seems to clear up my confusion. It appears that from his explanation we do indeed need class "scores", if I am reading his post correctly. $\endgroup$
    – aranglol
    Commented Apr 10, 2018 at 18:49
  • $\begingroup$ @Sycorax rephrased a bit, but basically I don't know where the OP got the idea that ROC needs probabilities in the first place. $\endgroup$
    – Calimo
    Commented Apr 10, 2018 at 19:42
  • $\begingroup$ @aranglol I hope it's clearer now. If you have Win and Loss you can't vary the threshold so you're out of luck. But if you have "Loss", "Null", "Win" and "Jackpot", then you're good to go. $\endgroup$
    – Calimo
    Commented Apr 10, 2018 at 19:45
  • $\begingroup$ I see what you mean now. I suppose when I stated: "From my understanding, do we not need the class "probabilities" (scores) produced from the model in order to graph the ROC curve in the first place" I put probabilities in quotations along with (scores) to note any sort of ranking systems (and not necessarily probabilities). I can see where I wasn't clear, sorry. The detection threshold being variable is very helpful though. The "probability" scores as reported by my model are necessary as this allows the variation in the threshold. Thanks Calimo. $\endgroup$
    – aranglol
    Commented Apr 10, 2018 at 21:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.