How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics?

Question

How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics?

Since in a binary classifier, there are two classes, often labelled positive (+1) and negative (-1). Yet, the classifier performance metrics [y] precision (PPV) and [x] recall (TPR) which are used to plot PR curves can have different values for each of the two classes (if you swap the positive and negative classes). In almost all examples of PR curves, there is usually only a single curve, when there should surely be at least two curves (one curve per class)?

More specifically:

1. Does the PR curve really only represent the precision and recall of a single class, or has some operation been done (e.g. averaging) to combine the precision and recall of both classes?

2. Does it make sense to judge a classifier's performance based on only looking at the PR curve for the positive class?

3. Why are the metrics TNR and NPV not somehow integrated into the curve or graph, for a better overview of classifier performance?

Answer 1

1. In my experience, yes, it does.

   Some libraries may give you a weighted average of the AUPRC across all of the classes, in addition to . For example, Weka includes a class that has a function to compute the weighted AUPRC. This function computes the AUPRC for each class individually, then averages them, weighting each class by the number of examples. Needless to say, the Weka class also includes a function for computing the AUPRC for each class individually, and by feeding this function some sample data it can be seen that the AUPRC is usually different for each class. I have also used the r package PRROC, which closely matches the AUPRC values for each class obtained by Weka.

---

2. It depends on the context. But yes, there are cases where it makes sense to primarily use the AUPRC of a given class when judging a classifier's performance.

   Ideally, you want to factor in the cost of the false positives when evaluating your classifier. Sometimes you can write them off, by saying that you can build the system to be "fail-safe", i.e., that there is not a large cost with a false positive. For example, if you have a tutoring system that buzzes harshly when a positive example is detected (of something like falling asleep, perhaps), then you probably want to make sure your system isn't detecting positive examples a lot when there aren't any (in other words, you care about your accuracy when detecting the negative class). Another example would be misdiagnosing a patient with cancer.

   In a fail-safe system, you can afford to have some number of false positives. For example, instead of a harsh buzz when a positive example is detected, you might just take note of the positive example, and add some extra lessons at the end of the session. The extra lessons aren't likely to bother the student as much as unnecessary harsh buzzes, so it's ok if some of the positive examples are actually negative examples (of not sleeping).

   Thus, if you can mitigate the cost of misclassifying the negative examples, you can focus on classifiers that are better at classifying the positive examples, even though they may not be as good at classifying the negative examples. This would mean you could focus on classifiers that have better a AUPRC for the positive class even if they have lower a AUPRC for the negative class. In my experience, classifiers with the same overall AUPRC can have a different AUPRC for each individual class. So, it could end up being a good idea to look at the AUPRC for a specific class.

---

3. Because TNR is not used to compute precision or recall. You would need to plot a different curve to incorporate TNR or NPV.

   That being said, you can get a better overview of a classifier's performance by aggregating the PRC for each class. An example would be this multi-class PRC in sci-kit learn. It includes a micro-averaged PRC in addition to the three curves that constitute it.

   Again, I think that the decision on which curve is most important to evaluate your classifier depends on how much you care about classifying each class correctly. If you care more about classes 1 and 2, then pick the classifier that performs best on those two, regardless of how it performs on class 3. If you focus on a classifier that performs best across all classifiers, it may actually classify classes 1 and 2 correctly fewer times than another classifier, even though it is better at classifying class 3.