22
$\begingroup$

It is easy to find a package calculating area under ROC, but is there a package that calculates the area under precision-recall curve?

Improve this question
$\endgroup$
2
  • $\begingroup$ ROCR, pROC - are really nice! $\endgroup$
    – Vladimir Chupakhin
    Commented Sep 5, 2011 at 15:37
  • 1
    $\begingroup$ They certainly are, yet AFAIK neither can calculate the area under precision-recall curve. $\endgroup$
    – user88
    Commented Sep 5, 2011 at 16:57

4 Answers 4

Reset to default
22
$\begingroup$

As of July 2016, the package PRROC works great for computing both ROC AUC and PR AUC.

Assuming you already have a vector of probabilities (called probs) computed with your model and the true class labels are in your data frame as df$label (0 and 1) this code should work:

install.packages("PRROC")

require(PRROC)
fg <- probs[df$label == 1]
bg <- probs[df$label == 0]

# ROC Curve    
roc <- roc.curve(scores.class0 = fg, scores.class1 = bg, curve = T)
plot(roc)

# PR Curve
pr <- pr.curve(scores.class0 = fg, scores.class1 = bg, curve = T)
plot(pr)

PS: The only disconcerting thing is you use scores.class0 = fg when fg is computed for label 1 and not 0.

Here are the example ROC and PR curves with the areas under them:

ROC Curve with AUC

PR Curve with AUC

The bars on the right are the threshold probabilities at which a point on the curve is obtained.

Note that for a random classifier, ROC AUC will be close to 0.5 irrespective of the class imbalance. However, the PR AUC is tricky (see What is "baseline" in precision recall curve).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Pls refer to the PRROC documentation linked in the answer above. $\endgroup$
    – arun
    Commented Oct 12, 2018 at 9:34
4
$\begingroup$

AUPRC() is a function in the PerfMeas package which is much better than the pr.curve() function in PRROC package when the data is very large. pr.curve() is a nightmare and takes forever to finish when you have vectors with millions of entries. PerfMeas takes seconds in comparison. PRROC is written in R and PerfMeas is written in C.

Improve this answer
$\endgroup$
0
2
$\begingroup$

A little googling returns one bioc package, qpgraph (qpPrecisionRecall), and a cran one, minet (auc.pr). I have no experience with them, though. Both have been devised to deal with biological networks.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This minet looked nice, but it needs to have some external adapter to make appropriate input from general data :-( $\endgroup$
    – user88
    Commented May 8, 2011 at 9:09
2
$\begingroup$

Once you've got a precision recall curve from qpPrecisionRecall, e.g.:

pr <- qpPrecisionRecall(measurements, goldstandard)

you can calculate its AUC by doing this:

f <- approxfun(pr[, 1:2])
auc <- integrate(f, 0, 1)$value

the help page of qpPrecisionRecall gives you details on what data structure expects in its arguments.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Doesn't the PR-curve require some more fancy integration? See: mnd.ly/oWQQw1 $\endgroup$
    – user88
    Commented Aug 31, 2011 at 12:37