Precision-Recall curve interpretation

Question

When given an example confusion matrix:

TP = 5000 FP = 1000

FN = 0 TN = 100

This would result in a point on the PR-curve where the x coordinate is fixed [1, y]

--> thus perfect recall --> meaning zero false negatives

--> however this says nothing about true negatives and false positives

---

When given an example confusion matrix:

TP = 5000 FP = 0

FN = 10 TN = 100

This would result in a point on the PR-curve where the y coordinate is fixed [x, 1]

--> thus perfect precision --> meaning no false positives

--> however this says nothing about false negatives and true negatives

---

When analysing a ROC curve it's intuitive to maximise TPR and minimise FPR. But for the PR-curve one needs to maximise both. How does one know to maximise recall vs maximising precision? When is recall more important than precision? How should you trade-off these 2 metrics?

Recall: TP/(TP+FN) How can you assume anything regarding this metric when you are interested in finding TN's?

Precision: TP/(TP+FP) Is it ok to assume precision is less important when you want to find TN's and thus 'value' points that are on the top and bottom left less? --> because you don't mind having false positives as long as you find true negatives.

Answer 1

There is no universal answer for how to optimize and it depends on the specific implementation and in particular, what are the consequences of FN vs. those of FP.

For example, the cost of an ad on a web site may be lower than that of making a marketing phone call. Hence, in the latter case you may require a higher precision when deciding whether to make a marketing call than when deciding whether to place an ad.

Answer 2

It may help to consider the questions precision and recall are trying to ask:

Recall: TP/(TP + FN)- Out of the instances that my model is interested in trying to predict accurately (TP + FN), how many is my model able to detect?

Precision: TP/(TP + FP)- Out of the instances that my model actually predicts as the class I am interested in (TP + FP), how many actually have the actual condition?

Which you are interested in optimizing depends a lot on the net costs of FP and FN (which itself is a function of the model you choses AND the prevalence of the condition you are interested in). If you are interested in something that summarizes both, the F-measure can be used, which is a harmonic mean of both.

F Measure = 2 x (Recall x Precision)/(Recall + Precision)

As an example, imagine you work for a car insurance company and you have a built two models that you can use to predict which of your customers is likely to get into an accident within the next year.

The first model has high recall and low precision; the second model has a low recall and high precision. This means two things: 1) Of the the individuals who get into a car accident, the 1st model is able to predict a higher proportion than the second model (recall). 2) Of the individuals models predict getting into an accident, the second model achieves a higher proportion (precision).

Note: if you value a high specificity, i.e. TNR, of your model, you may or may not want a high precision: precision is technically about the positive class, but its complement (FP/(TP+ FP)), if low, may imply a high TNR. But that is not always the case.