I created model (logistic regression) and now trying to create Precision-Recall plot and calculate area under Precision-Recall Plot. I'd like to note that this model is defective:

glm.fit: fitted probabilities numerically 0 or 1 occurred

Data can be downloaded here: download

The problem is that I am getting different results from different libraries.

Manual calculations of AUC and library ROCR:

data = read.csv("lr.csv", stringsAsFactors = F)
predictions = data$predictions
actual = data$actual
predict_obj = prediction(predictions, actual)
prc = performance(predict_obj, "prec", "rec")
x = [email protected][[1]]
y = [email protected][[1]]

nans = is.nan(y)  
if (sum(nans) > 0) {
  y = y[!nans]
  x = x[!nans]

plot(prc, main = "ROCR library")  
auc_prc = sum(diff(x) * (head(y, -1) + tail(y, -1)) / 2)

gives me this plot and AUC = 0.3630107:

enter image description here

with cutoffs:

enter image description here

library PRROC:

pr_curve = pr.curve(scores.class0 = predictions[actual == 1],  scores.class1 = predictions[actual == 0], curve = T)
plot(pr_curve, main = "PRROC library")

gives me this plot and AUC = 0.7160321:

enter image description here

library PerfMeas:

obj = precision.at.all.recall.levels(predictions, actual)
plot(obj$recall, obj$precision, type = "l", main = "PerfMeas library")
AUPRC(list(precision.at.all.recall.levels(predictions, actual)), comp.precision = TRUE)

gives me this plot and AUC = 0.6938177:

enter image description here

library DMwR (as wrapper of ROCR):

PRcurve(predictions, actual, main = "DMwR library")

gives me this plot (without AUC):

enter image description here

My question is: why plots are so different? Which graph should I choose and what is my AUC?

This question is connected with my another question: What is actually AUC (area under the curve)?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.