40
$\begingroup$

I read some definitions of recall and precision, though it is every time in the context of information retrieval. I was wondering if someone could explain this a bit more in a classification context and maybe illustrate some examples. Say for example I have a binary classifier which gives me a precision of 60% and a recall of 95%, is this a good classifier ?

Maybe to help my goal a bit more, what is the best classifier according to you ? (dataset is imbalanced. Majority class has twice the amount of examples of the minority class)

I'd personally say 5 because of the area under the receiver operator curve.

(as you can see here model 8 has a low precision, very high recall, but one of the lowest AUC_ROC, does that make it a good model ? or a bad one ?)

enter image description here


edit:

I have an excel file with more information: https://www.dropbox.com/s/6hq7ew5qpztwbo8/comparissoninbalance.xlsx

In this document the area under the reciever operator curve can be found and the area under the precision recall curve. Together with the plots.

$\endgroup$
  • 5
    $\begingroup$ Why the negative vote? This seems like an interesting question (certainly one several people thought possible to answer). $\endgroup$ – Gala Jun 26 '13 at 11:02
  • $\begingroup$ How many test cases were involved in calculating the performance measures? Also, if you compare several classifiers on the same data set, you can use a paired design (there are several questions & answers around on CV discussing classifier comparison). $\endgroup$ – cbeleites Jun 26 '13 at 12:17
  • $\begingroup$ I have 799 samples (if that is what you mean by test case). Everytime I split the data in 10 subset. 1 subset for testing the classifiers and the rest for training the classifiers (The classifiers which are compared are trained on the same data). $\endgroup$ – Olivier_s_j Jun 26 '13 at 12:45
  • $\begingroup$ Do the classifiers output a binary value, or do they output a real value, and then use a threshold to make it binary ? $\endgroup$ – galath Jun 26 '13 at 12:48
  • $\begingroup$ They output binary output. $\endgroup$ – Olivier_s_j Jun 26 '13 at 12:53
38
$\begingroup$

Whether a classifier is “good” really depends on

  1. What else is available for your particular problem. Obviously, you want a classifier to be better than random or naive guesses (e.g. classifying everything as belonging to the most common category) but some things are easier to classify than others.
  2. The cost of different mistakes (false alarm vs. false negatives) and the base rate. It's very important to distinguish the two and work out the consequences as it's possible to have a classifier with a very high accuracy (correct classifications on some test sample) that is completely useless in practice (say you are trying to detect a rare disease or some uncommon mischievous behavior and plan to launch some action upon detection; Large-scale testing costs something and the remedial action/treatment also typically involve significant risks/costs so considering that most hits are going to be false positives, from a cost/benefit perspective it might be better to do nothing).

To understand the link between recall/precision on the one hand and sensitivity/specificity on the other hand, it's useful to come back to a confusion matrix:

                      Condition: A             Not A

  Test says “A”       True positive (TP)   |   False positive (FP)
                      ----------------------------------
  Test says “Not A”   False negative (FN)  |    True negative (TN)

Recall is TP/(TP + FN) whereas precision is TP/(TP+FP). This reflects the nature of the problem: In information retrieval, you want to identify as many relevant documents as you can (that's recall) and avoid having to sort through junk (that's precision).

Using the same table, traditional classification metrics are (1) sensitivity defined as TP/(TP + FN) and (2) specificity defined as TN/(FP + TN). So recall and sensitivity are simply synonymous but precision and specificity are defined differently (like recall and sensitivity, specificity is defined with respect to the column total whereas precision refers to the row total). Precision is also sometimes called the “positive predictive value” or, rarely, the “false positive rate” (but see my answer to Relation between true positive, false positive, false negative and true negative regarding the confusion surrounding this definition of the false positive rate).

Interestingly, information retrieval metrics do not involve the “true negative” count. This makes sense: In information retrieval, you don't care about correctly classifying negative instances per se, you just don't want too many of them polluting your results (see also Why doesn't recall take into account true negatives?).

Because of this difference, it's not possible to go from specificity to precision or the other way around without additional information, namely the number of true negatives or, alternatively, the overall proportion of positive and negative cases. However, for the same corpus/test set, higher specificity always means better precision so they are closely related.

In an information retrieval context, the goal is typically to identify a small number of matches from a large number of documents. Because of this asymmetry, it is in fact much more difficult to get a good precision than a good specificity while keeping the sensitivity/recall constant. Since most documents are irrelevant, you have many more occasions for false alarms than true positives and these false alarms can swamp the correct results even if the classifier has impressive accuracy on a balanced test set (this is in fact what's going on in the scenarios I mentioned in my point 2 above). Consequently, you really need to optimize precision and not merely to ensure decent specificity because even impressive-looking rates like 99% or more are sometimes not enough to avoid numerous false alarms.

There is usually a trade-off between sensitivity and specificity (or recall and precision). Intuitively, if you cast a wider net, you will detect more relevant documents/positive cases (higher sensitivity/recall) but you will also get more false alarms (lower specificity and lower precision). If you classify everything in the positive category, you have 100% recall/sensitivity, a bad precision and a mostly useless classifier (“mostly” because if you don't have any other information, it is perfectly reasonable to assume it's not going to rain in a desert and to act accordingly so maybe the output is not useless after all; of course, you don't need a sophisticated model for that).

Considering all this, 60% precision and 95% recall does not sound too bad but, again, this really depends on the domain and what you intend to do with this classifier.


Some additional information regarding the latest comments/edits:

Again, the performance you can expect depends on the specifics (in this context this would be things like the exact set of emotions present in the training set, quality of the picture/video, luminosity, occlusion, head movements, acted or spontaneous videos, person-dependent or person-independent model, etc.) but F1 over .7 sounds good for this type of applications even if the very best models can do better on some data sets [see Valstar, M.F., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012). Meta-analysis of the first facial expression recognition challenge. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42 (4), 966-979.]

Whether such a model is useful in practice is a completely different question and obviously depends on the application. Note that facial “expression” is itself a complex topic and going from a typical training set (posed expressions) to any real-life situation is not easy. This is rather off-topic on this forum but it will have serious consequences for any practical application you might contemplate.

Finally, head-to-head comparison between models is yet another question. My take on the numbers you presented is that there isn't any dramatic difference between the models (if you refer to the paper I cited above, the range of F1 scores for well-known models in this area is much broader). In practice, technical aspects (simplicity/availability of standard libraries, speed of the different techniques, etc.) would likely decide which model is implemented, except perhaps if the cost/benefits and overall rate make you strongly favor either precision or recall.

$\endgroup$
  • $\begingroup$ My classifier classifies faces into positive or negative emotion. I ran a couple of classification algorithms with 10 fold cross validation and I even get 100% recall sometimes, though the precision is for all the classifiers almost the same (around 65%). I work with an imbalanced dataset (majority class has twice the amount of samples of the minority class). I therefore thought this probably meant my models weren't learning the pattern. $\endgroup$ – Olivier_s_j Jun 26 '13 at 11:09
  • 1
    $\begingroup$ @Gala: What's the difference between a 'false alarm' and a 'false negative'? Do you mean a 'false alarm' and a 'false positive'? $\endgroup$ – Matt O'Brien Jun 25 '15 at 4:54
  • $\begingroup$ @MattO'Brien A false alarm is a false positive (i.e. the system says “Beware!” or “Found something!” but there is nothing and no need to act, hence “false alarm”). Maybe the terminology is a bit confusing, I will review the answer later. $\endgroup$ – Gala Jun 25 '15 at 5:56
  • $\begingroup$ @Ojtwist If positive has prior probability 67%, then you can get 100% recall and ~67% precision by predicting everything as positive. $\endgroup$ – Mark Oct 18 '16 at 20:55
21
$\begingroup$

In the context of binary classification, examples are either positive or negative.

  • The recall addresses the question: "Given a positive example, will the classifier detect it ?"
  • The precision addresses the question: "Given a positive prediction from the classifier, how likely is it to be correct ?"

So it depends if the focus is on positive examples or on positive predictions.

One could say "at a recall at least 90%, the classifier with the highest precision is 4." But if what matters is the quality of the predictions, among the classifiers with a precision of at least 70%, the one that achieves highest recall is 6.

$\endgroup$
6
$\begingroup$

Moving from continuous predictions, as used in computing ROC area (concordance probability; c-index) to a discontinuous improper scoring rule (forced-choice classification accuracy) results in all kinds of anomalies and will mislead the analyst to choose the wrong predictors and/or the wrong model. It is much better to make decisions on the basis of proper scoring rules (e.g., log-likelihood/deviance/logarithmic probability scoring rule; Brier score (quadratic probability accuracy score)). Among the many advantages of such an approach is the identification of observations for which classification is hazardous to your health due to uncertainty caused by mid-range probabilities.

$\endgroup$
  • $\begingroup$ Aren't these scoring rules only applicable when the model gives probabilistic outputs ? $\endgroup$ – Olivier_s_j Jun 26 '13 at 14:38
  • 3
    $\begingroup$ You mentioned ROC AUC which requires probabilistic outputs (or something that is a monotonic function of same). The best predictive methods have probabilities under the hood. $\endgroup$ – Frank Harrell Jun 26 '13 at 15:12
  • $\begingroup$ P.S. Somers' $D_{xy}$ rank correlation between a continuous or ordinal predictor (including an overall predicted probability) and binary $Y$ is a simple linear translation of ROC area. It is a very useful measure whether you've converted the predictor to a probability or not. $\endgroup$ – Frank Harrell Jun 26 '13 at 20:50
  • $\begingroup$ The problem with using ROC and non "forced-choice" classification accuracy measures is that they don't actually mean anything in the context of application where one must actually choose a threshold according to a business case. For example in a financial crime detection product for investment banking a business may only have the human resources to investigate 100 classifications a year, this induces a fixed threshold which should optimized against. Gini, ROC area, etc won't tell you how good the model is at this threshold. ... $\endgroup$ – samthebest Aug 22 '14 at 9:37
  • 1
    $\begingroup$ It is very reasonable to have a budgetary limit on the number of investigations. This requires no thresholds, no new optimization, no classifications, and no ROC analysis. One merely sorts the observations in descending order of predicted risk and chooses the first 100 observations to investigate. This is related to a lift curve used in marketing. $\endgroup$ – Frank Harrell Aug 22 '14 at 12:41
5
$\begingroup$

Neither precision nor recall tell the full story, and it is hard to compare a predictor with, say, 90% recall and 60% precision to a predictor with, say, 85% precision and 65% recall - unless, of course, you have cost/benefit associated with each of the 4 cells (tp/fp/tn/fn) in the confusion matrix.

An interesting way to get a single number (proficiency, aka uncertainty coefficient) describing the classifier performance is to use information theory:

proficiency = I(predicted,actual) / H(actual)

i.e., it tells you what fraction of the information present in the actual data was recovered by the classifier. It is 0 if either precision or recall is 0 and it is 100% if (and only if) both precision and recall are 100%. In that it is similar to F1 score, but proficiency has a clear information-theoretical meaning while F1 is just a harmonic average of two numbers with a meaning.

You can find paper, presentation and code (Python) to compute the Proficiency metric here: https://github.com/Magnetic/proficiency-metric

$\endgroup$
  • $\begingroup$ interesting! can you give us an example of how to calculate proficiency? cause the formula looks odd to me. $\endgroup$ – lonesome Oct 11 '16 at 12:58
  • $\begingroup$ Please see github.com/Magnetic/proficiency-metric $\endgroup$ – sds Oct 11 '16 at 13:09
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Scortchi Oct 21 '16 at 11:40

protected by Glen_b Feb 8 '17 at 9:56

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.