12
$\begingroup$

What is the best way to explain why $\text{Precision} + \text{Recall}$ is not a good measure, say, compared to F1?

$\endgroup$
4
  • $\begingroup$ What would it mean? How would you interpret it? What would it actually tell you? $\endgroup$ – Matthew Drury Jun 22 '17 at 23:45
  • 1
    $\begingroup$ You should change the title by replacing "Precision+Recall" by "the sum of Precision and Recall" to make it clearer what you want. $\endgroup$ – g3o2 Jun 23 '17 at 8:56
  • $\begingroup$ @g3o2 are we talking grammar here, or am I missing something greater? $\endgroup$ – matt Jun 24 '17 at 17:12
  • $\begingroup$ Not really, just noting that it can also be read Precision & Recall, in particular when reading the title only. $\endgroup$ – g3o2 Jun 24 '17 at 22:24
18
$\begingroup$

It's not that $\text{Precision} + \text{Recall}$ is a bad measure per se, its just that, on its own, the resulting number doesn't represent anything meaningful. You are on the right track though... what we are looking for is a combined, average of the two performance measures since we don't want to have to choose between them.

Recall that precision and recall are defined as:

$$\text{Precision} = \frac{\text{True Positive}}{\text{Predicted Positive}}$$ $$\text{Recall} = \frac{\text{True Positive}}{\text{Actual Positive}}$$

Since they both have different denominators, adding them together results in something like this: $$\frac{\text{True Positive}\left(\text{Predicted Positive}+\text{Actual Positive}\right)}{\text{Predicted Positive}\times \text{Actual Positive}}$$ ... which isn't particularly useful.

Lets go back to adding them together, and make a tweak: multiply them by $\frac{1}{2}$ so that they are the stay in the correct scale, $[0-1]$. This is taking the familiar average of them.

$$ \frac{1}{2} \times \left( \frac{\text{True Positive}}{\text{Predicted Positive}} + \frac{\text{True Positive}}{\text{Actual Positive}} \right) $$

So, we have two quantities, which have the same numerator, but different denominators and we would like to take the average of them. What do we do? Well we could flip them over, take their inverse. Then you could add them together. So they are "right side up", you take the inverse again.

This process of inverting, and then inverting again turns a "regular" mean into a harmonic mean. It just so happens that the harmonic mean of precision and recall is the F1-statistic. The harmonic mean is generally used instead of the standard arithmetic mean when dealing with rates, as we doing are here.

In the end, the F1-statistic is just the average of precision and recall, and you use it because you don't want to choose one or the other to evaluate the model's performance.

$\endgroup$
2
  • 2
    $\begingroup$ Really many thanks for kindly developing the harmonic mean from the algebraic mean! but what probably doesn't sit very firmly with me, is the part where you say "which isn't particularly useful". In that vein I've commented below on the other two answers now. Just in case you'd take it one step further.. E.g. imagine I wish to pick the best classifier among a group of classifiers tested over the same dataset. $\endgroup$ – matt Jun 23 '17 at 5:22
  • $\begingroup$ @matt, using any combined measure will bring your model choice to a certain point but not beyond. Two models having the same F1 value may show completely opposite Recall and Precision values. Therefore, for F1 being the same, you will have to choose between Recall and Precision. $\endgroup$ – g3o2 Jun 23 '17 at 11:28
5
$\begingroup$

The short answer is: you would not expect the summing of two percentages which have two different denominators to have any particular meaning. Hence, the approach to take an average measure such as F1, F2 or F0.5. The latter retain at least the property of a percentage. What about their meaning though?

The beauty of Precision and Recall as separate measures is their ease of interpretation and the fact that they can be easily confronted with the model's business objectives. Precision measures the percentage of true positives out of the cases classified as positive by the model. Recall measures the percentage of true positives found by the model out of all the true cases. For many problems, you will have to choose between optimizing either Precision or Recall.

Any average measure looses the above interpretation and boils down to which measure you prefer most. F1 means either you don't know whether you prefer Recall or Precision, or you attach equal weight to each of them. If you consider Recall more important than Precision, then you should also allocate a higher weight to it in the average calculation (e.g F2), and vice versa (e.g F0.5).

$\endgroup$
4
$\begingroup$

Adding the two is a bad measure. You'll get a score of at least 1 if you flag everything as positive, since that's a 100% recall by definition. And you'll get a little precision bump on top of that. The geometric mean used in F1 emphasizes the weak link, since it is multiplicative; you have to at least do okay with both precision and recall to get a decent F1 score.

$\endgroup$
3
  • $\begingroup$ Actually, it is this exact emphasis of the weak link, which I have found superfluous when Precision and Recall are both sensible rather than edgy. When they are both not edgy, I am not sure I see the added value of a metric emphasizing the similarity between them, or put differently penalizing by the size of their difference. This exact property has in part motivated my original question here. $\endgroup$ – matt Jun 23 '17 at 5:23
  • $\begingroup$ Sounds like extra work to me. If you value a percentage point of recall just as much as one of precision, then I guess your measure is the one to use. But I can't image you would. Recall is probably going to dominate, even if you reduce the ranges. You could scale recall to be apples-to-apples with precision, but that's again more work and makes the interpretation less clear. $\endgroup$ – Ben Ogorek Jun 23 '17 at 13:14
  • 1
    $\begingroup$ Not sure why assume recall should dominate (?) but scaling recall to be apples-to-apples with precision might be an interesting related discussion here or elsewhere ― a pointer in the right direction might be a nice to have :) and otherwise thanks again $\endgroup$ – matt Jun 24 '17 at 17:24
3
$\begingroup$

F1 score is especially valuable in case of severely asymmetric probabilities.

Consider the following example: we test for a rare but dangerous illness. Let's assume that in a city of 1.000.000 people only 100 are infected.

Test A detects all these 100 positives. However, it also has 50% false positive rate: it erroneously shows another 500.000 people to be ill.

Meanwhile, test B misses 10% of the infected, but gives only 1.000 false positives (0.1% false positive rate)

Let's calculate the scores. For test A, precision will be effectively 0; recall will be exactly 1. For test B, precision will still be rather small, about 0.01. Recall will be equal to 0.9.

If we naively sum or take arithmetic mean of precision and recall, this will give 1 (0.5) for test A and 0.91 (0.455) for test B. So, test A would seem marginally better.

However, if we look from a practical perspective, test A is worthless: if a person is tested positive, his chance to be truly ill is 1 in 50.000! Test B has more practical significance: you may take 1.100 people to the hospital and observe them closely. This is accurately reflected by F1 score: for test A it will be close to 0.0002, for test B: (0.01 * 0.9) / (0.01 + 0.9) = 0.0098, which is still rather poor, but about 50 times better.

This match between score value and practical significance is what makes F1 score valuable.

$\endgroup$
2
  • $\begingroup$ Thanks. Maybe I've not sufficiently immersed myself in the case, but does this elucidation not hinge on the pragmatic advantage of allocating resources to the "positives" in a real-life domain where detecting one result (a positive) is the goal? this is not always the case, that the goal is detecting one result is it? sometimes you just want to know whether it's an apple or a pair, and both types of error have the same practical real-world cost. $\endgroup$ – matt Jun 24 '17 at 17:52
  • $\begingroup$ Above all, what I fail to see is how this property of being "better" scales to cases where the (absolute) difference between precision and recall is less pathological. Maybe the intuition is inherently there, but I am not there yet... $\endgroup$ – matt Jun 24 '17 at 17:53
2
$\begingroup$

In general, maximizing the geometric mean emphasizes the values being similar. For example, take two models: the first has (precision, recall) = (0.8, 0.8) and the second has (precision, recall) = (0.6, 1.0). Using the algebraic mean, both models would be equivalent. Using the geometric mean, the first model is better because it doesn't trade precision for recall.

$\endgroup$
7
  • 1
    $\begingroup$ Thanks a lot. However, in practical terms, I fail to see any universally applicable preference between e.g. (0.8, 0.8) and (0.7, 0.9). Maybe you had hinted at something deeper in "trading off Precision for recall" ― which I am not picking up myself (yet). For me algebraically averaging two types of error, simply gives the simplest average of them, without any bias to similarity. E.g. I might use the simple summation of Precision and Recall to figure which of two classifiers gives me less error. $\endgroup$ – matt Jun 23 '17 at 5:24
  • $\begingroup$ We can take this to an extreme. Let's say you have one system that has a (precision, recall) = (0.6, 0.6). That means that when it says "yes" its right 60% of the time and it correctly catches 60% of "yes" events. Now let's compare this to a system that has (0.3, 1). This has a better algebraic mean, but what is it doing? It is catching all the "yes" events, but it's also saying "yes" incorrectly a lot. Is that good? Is that bad? It depends on why you are building the system. What action will you take when you see a "yes" prediction? What is the consequence of missing a "yes" event? $\endgroup$ – roundsquare Jun 24 '17 at 15:11
  • 1
    $\begingroup$ None of these measures are proper accuracy scoring rules $\endgroup$ – Frank Harrell Jun 24 '17 at 16:11
  • $\begingroup$ @roundsquare many thanks, but for non-pathological cases ― where both are not near 0 and 1 ― I probably need help seeing the benefit of emphasizing similarity between the two, in the final measure! $\endgroup$ – matt Jun 24 '17 at 17:19
  • $\begingroup$ @FrankHarrell thanks for pointing at "the elephant in the room" $\endgroup$ – matt Jun 24 '17 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.