Explain intuitively why do we need f1, precision and recall

Question

Imagine you are trying to explain the purpose of f1, precision and recall to a person? How would you proceed on why do we need such measures instead of simply computing incorrect / correct ratio?

I tried traditional explanation with false/true positives/negatives but failed because the person brought up the following example.

If I have 5 blue balls and 5 red balls and I create a classifier to predict which balls are red and which balls are blue. My classifier starts prediction and predicts 4 blue balls and 6 red balls. Well, this is easy and I can say that it's accuracy is 90%. Let's try another example, 99 blue balls and 1 red ball. My classifier predicts 100 blue balls, oh well its accuracy is 99%, very good. If I have 1000 blue balls and my classifier predicts 800 as blue and 200 as red, well 80% is still not bad. So what's the point of having all these confusing false positives/negatives when you can simply calculate the number of correct and incorrect guesses?

Answer 1

F1, precision and recall aren't really relevant to classification problems with equivalent and equally prevalent classes, such as "blue" vs "red" in your example, when you care as much about a red ball being mis-classified as blue as you do the other way around. In that case you would indeed just use the overall accuracy, as you suggested.

These scores are important, though, when one class can be described as "positive", and the other as "negative". A typical example would be a medical diagnostic test, which has to discriminate between sick and healthy people. The precision in that case measures the % of people diagnosed "sick" who really are sick, and the recall measures the % of people who are sick, and correctly identified by the test as such. In a medical context, you might place especially high importance on the recall score, since it's typically worse to miss a sick person than to get a false positive on a healthy person.

Another reason to use precision & recall is when you have skewed classes. If 99% of your examples belong in one class, any classifier can easily achieve 99% overall accuracy by always predicting that class label. But as a percentage of the examples in the rare class, it gets 100% wrong, so the classifier's recall for that class would be 0, and its precision undefined (0/0). If the purpose of the classifier is to pick out cases of the rare class, these scores tell you it does a very poor job, while the accuracy tells you nothing.

(Edit: as pointed out by user7019377, the above assumes that you care about the individual class accuracies and want to optimize those rather than the overall accuracy. If you have skewed classes but you only care about overall accuracy, then once again precision & recall don't matter. Also I might add that you don't necessarily need to compute precision & recall to account for class imbalances - you can also just compute a regular accuracy score separately in each class (and potentially average those accuracy scores).)

These two reasons often go hand in hand. E.g. in the medical context sick people are rare relative to healthy people (esp. for a particular disorder being tested), so I could get very high accuracy by designing a test that just returns "healthy" all the time, but that accuracy would be misleading about the usefulness of the test, and potentially many people would suffer for it.

Answer 2

Here is one possible generalisation on how to choose the appropriate metric for classification problem. Basically, we want a metric that

• the higher the better. Possibly (and more preferably???) bounded but I don't have a convincing argument why it should be, but I believe it's not necessary to have something blows up to infinity.

• takes into account all possible cases: TP, FP, FN, TN

So we may naturally come up with a metric of the following form: $$F = \frac{a_1 TP + b_1 TN}{a_2 TP + c_2 TN + d_2 FP + b_2 FN} = \frac{\text{Weighted number of SOME correct decisions}}{\text{Weighted number of ALL decisions}}$$

where $a_1 \leq a_2, b_1 \leq b_2$ and $a_i,b_i,c_i,d_i \geq 0$. This metric is bounded between $[0,1]$. It is not hard to see that F1 score, accuracy, precision, recall are just special cases of this metric, for example:

$$Accuracy = \frac{TP+TN}{TP+TN+FP+FN} \text{ (where } a_i = b_i = c_i = d_i = 1)$$

$$F_{\beta} = \frac{(\beta^2 + 1) TP}{(\beta^2 + 1) TP + \beta^2 FP + FN} \text{ (where } a_1 = a_2 = \beta^2 + 1, b_1 = 0, b_2 = 1, c_2 = 0, d_2 = \beta^2)$$

where if $\beta \in [0, \infty]$. This is called F-measure and when $\beta = 1$, you recover the F1 score.

So my general answer to your question is: the choice of the coefficients depends totally on how you evaluate the relative importance of one case versus others. Your evaluation depends on other external factors, for example data structure (balanced or unbalanced), or domain-specific (spam filtering, biomedicine).