2
$\begingroup$

The F1 score is commonly defined as the harmonic mean of precision and recall, which is equivalent to:

$$ \text{F1 score} = \frac{2 \times \mathit{TP}}{ 2 \times \mathit{TP} + \mathit{FP} + \mathit{FN} } $$

This is a good measure of how well the algorithm is at picking out relevant items from a set. However, I am also interested in how good the algorithm is at leaving irrelevant items in the set, and therefore, I feel that the number of true negatives should be considered.

To do this, I'm considering calculating an F1 score where positives are treated as negatives and vice-versa. I would call this the "opposing F1 score". It would be calculated like this:

$$ \text{opposing F1 score} = \frac{ 2 \times \mathit{TN} }{ 2 \times \mathit{TN} + \mathit{FN} + \mathit{FP} } $$

I could also calculate the opposing precision and recall values like this:

$$ \text{opposing precision} = \frac{\mathit{TN}}{\mathit{TN} + \mathit{FN}} $$ $$ \text{opposing recall} = \frac{\mathit{TN}}{\mathit{TN}+\mathit{FP}} $$

Then I could train the algorithm to achieve a maximal value of the average (for some definition of average) of the F1 score and the opposing F1 score, perhaps.

Is anybody else doing this? Are there better ways of achieving my aims?

Context:

The problem I'm working on is not the usual problem for these measures. The task is to create a subset of items where all duplicates are excluded. I am interested in both excluding the greatest number of duplicates, and in including the greatest number of non-duplicates. So far, I have been using the measures for precision, recall and F1 score on pairs of items (so a true positive would be when the algorithm correctly classifies a pair as a duplicate). But as you can see, these measure aren't perfectly suited for my use-case.

$\endgroup$
1
$\begingroup$

Although the two measures have quite different logic, the F1 score is actually a special case of the specific agreement coefficient. Specifically, F1 is equivalent to specific agreement for the positive class when there are two raters, two categories, and no missing data. The "opposing F1 score" that you propose is equivalent to specific agreement for the negative class in this same scenario.

I encourage you to investigate the specific agreement formulation, as it is more flexible than the F1 score (in that it can handle multiple raters, multiple categories, and missing data) and has an nice interpretation in addition to the harmonic mean of precision and recall: specific agreement is equal to the probability that a randomly selected rater will assign a given item to a category given that another randomly selected rater has also assigned that item to that category.

$\endgroup$
0
$\begingroup$

The closest I got to an answer on this was the approach taken by the precision_recall_fscore_support function and the classification_report function in scikit-learn.

By default, the precision, recall, F1 score and support of each label is returned. That means that if you have only two labels, where one label is the opposite of the other label, then one F1 score is going to be the "opposing F1 score" of the other.

>>> from sklearn.metrics import precision_recall_fscore_support
>>> import numpy as np
>>> y_true = np.array(["dupe", "diff", "diff", "diff", "dupe", "diff"])
>>> y_pred = np.array(["dupe", "dupe", "diff", "diff", "diff", "diff"])
>>> precision_recall_fscore_support(y_true, y_pred, labels=["dupe", "diff"])
(array([ 0.5 ,  0.75]), array([ 0.5 ,  0.75]), array([ 0.5 ,  0.75]), array([2, 4]))
>>> print classification_report(y_true, y_pred)
             precision    recall  f1-score   support

       diff       0.75      0.75      0.75         4
       dupe       0.50      0.50      0.50         2

avg / total       0.67      0.67      0.67         6

So, how can we combine them to create an average? There are multiple ways. classification_report uses a weighted average, and precision_recall_fscore_support lets you choose between the averaging methods micro, macro, samples, weighted.

Update: This led me to find this interesting draft paper: Macro- and micro-averaged evaluation measures by Vincent Van Asch.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.