Skip to main content
3 of 3
added 1 character in body
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

ROC and PR curves are primarily visual tools to select a given operating point (cutoff) of existing models for the task at hand (high recall, precision, ...). Area under either curve is a commonly used, though not very intuitive, summary statistic of the performance of models over their entire operating range.

In my opinion, AUC is useful as a scoring function in hyperparameter search, or for studies that compare the performance of different learning methods (e.g. where you don't want to examine a specific operating point). That said, AUC is far less practical to finally select a model to be used for a specific task and it is entirely irrelevant to select the model's operating point. It is perfectly possible for a suitable model for a given task to have terrible AUC under either curve.


Example

Suppose I want very high precision but don't need a lot of recall and I have two models $A$ and $B$ with corresponding PR curves (precision as function of recall): $$ \begin{align} PR_A(r) &= e^{-2r} \quad \rightarrow \quad \text{PR-AUC} \approx 0.43\\ PR_B(r) &= 0.80 \quad \rightarrow \quad \text{PR-AUC} = 0.80 \end{align} $$ The precision of model $A$ is higher than $B$ up to about $11\%$ recall. Its AUC is far worse, though. Given the application I presented, model $A$ is probably the best choice. Examples of such applications:

  • gene prioritization: rank genes based on association with diseases; the top ranked genes are likely to become targets for (expensive) biological analyses.
  • fraud detection: rank transactions based on potential of fraudulence; top ranked transactions may lead to lawsuits, false positives lead to counterclaims.
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76