Confusion between precision and recall

Question

From what I understand, precision and recall are calculated by:

Precision = TP/(TP + FP) Recall = TP/(TP + FN)

But how do we actually interpret these two scores? For example, let's say we have some classification model which predicts whether a customer will end up making a purchase (positive class) or not, and I'd like to choose a performance metric. Shouldn't recall be more important in such a case since there is a higher cost associated with false negatives (i.e. we don't want to misclassify customers who actually make a purchase as not making one)?

Answer 1

Yes, recall should be more important, since its denominator only involves subjects who truly have disease (includes FN, which have disease but have bad prediction). Precision's denominator includes truly diseased plus FP, who don't have disease).

Recall is also known as sensitivity, and precision is known as predictive value positive (PV+), which depends on prevalence of disease.

Accuracy is the number correctly predicted out of all subjects: \begin{equation} Acc=(TP+TN)/(TP+FP+TN+FN) \end{equation}

You can use the F1 score which is based on precision and recall.

Rather, I like sens and spec, and ROC-AUC (receiver operator characteristic curve - area under the curve).

When trying to get a clinical diagnostic test approved by the FDA, which a hospital can be reimbursed for by medical insurance (in the U.S.), FDA requires at least 95% sensitivity (recall) and high specificity (>90%). But it's very difficult to achieve high levels if specificity, so you can only hope they approve your test at 90%+ spec. To over come the lower spec, you can run the test twice (which is a classic examination question for graduate students in statistics, i.e., "here are the test results for a test run twice, what's the sens and spec?")

A problem with recall (sens) is that it can be high (>90%) for a lot of diagnostic tests (classifiers). However, not true for spec, which is almost always lower than sens.

Answer 2

You are completely correct that the costs of false positives and false negatives are different. However, note that neither precision nor recall use any estimate of cost! Whether your costs differ by a factor of 2, 5, 10 or 100 will have zero impact on precision and recall.

Thus, I recommend that you use probabilistic classifications: how likely is a particular customer to make a purchase? You can then use a threshold on these predicted probabilities to make a further decision. After all, you might make more than two different decisions depending on the customer, e.g., using different incentives.

More information can be found at Classification probability threshold. Note that precision and recall (and many other KPIs) suffer from the same problems as accuracy: Why is accuracy not the best measure for assessing classification models? and Is accuracy an improper scoring rule in a binary classification setting?