It is easy to find a package calculating area under ROC, but is there a package that calculates the area under precision-recall curve?
As of July 2016, the package PRROC works great for computing both ROC AUC and PR AUC.
Assuming you already have a vector of probabilities (called
probs) computed with your model and the true class labels are in your data frame as
df$label (0 and 1) this code should work:
install.packages("PRROC") require(PRROC) fg <- probs[df$label == 1] bg <- probs[df$label == 0] # ROC Curve roc <- roc.curve(scores.class0 = fg, scores.class1 = bg, curve = T) plot(roc) # PR Curve pr <- pr.curve(scores.class0 = fg, scores.class1 = bg, curve = T) plot(pr)
PS: The only disconcerting thing is you use
scores.class0 = fg when
fg is computed for label 1 and not 0.
Here are the example ROC and PR curves with the areas under them:
The bars on the right are the threshold probabilities at which a point on the curve is obtained.
Note that for a random classifier, ROC AUC will be close to 0.5 irrespective of the class imbalance. However, the PR AUC is tricky (see What is "baseline" in precision recall curve).
AUPRC() is a function in the
PerfMeas package which is much better than the
pr.curve() function in
PRROC package when the data is very large.
pr.curve() is a nightmare and takes forever to finish when you have vectors with millions of entries.
PerfMeas takes seconds in comparison.
PRROC is written in R and
PerfMeas is written in C.
Once you've got a precision recall curve from
pr <- qpPrecisionRecall(measurements, goldstandard)
you can calculate its AUC by doing this:
f <- approxfun(pr[, 1:2]) auc <- integrate(f, 0, 1)$value
the help page of
qpPrecisionRecall gives you details on what data structure expects in its arguments.