9
$\begingroup$

The Scikit-Learn package in Python has two metrics: f1_score and fbeta_score. Each of these has a 'weighted' option, where the classwise F1-scores are multiplied by the "support", i.e. the number of examples in that class.

Is there any existing literature on this metric (papers, publications, etc.)? I can't seem to find any.

$\endgroup$
2
  • $\begingroup$ Did you find any reference or how the F1-Score is calculated. $\endgroup$
    – khan
    Mar 16, 2018 at 18:55
  • $\begingroup$ Does the update answer your question? $\endgroup$ Jun 22, 2018 at 5:05

1 Answer 1

11
$\begingroup$

The F1 Scores are calculated for each label and then their average is weighted by support - which is the number of true instances for each label. It can result in an F-score that is not between precision and recall.

For example, a simple weighted average is calculated as:

>>> import numpy as np;
>>> from sklearn.metrics import f1_score
>>> np.average( [0,1,1,0 ], weights=[1,1,1,1] )
0.5
>>> np.average( [0,1,1,0 ], weights=[1,1,2,1] )
0.59999999999999998
>>> np.average( [0,1,1,0 ], weights=[1,1,4,1] )
0.7142857142857143

The weighted average for each F1 score is calculated the same way:

f_score = np.average(f_score, weights=weights)

For example:

>>> f1_score( [1,0,1,0], [0,0,1,1] )
0.5
>>> f1_score( [1,0,1,0], [0,0,1,1], sample_weight=[1,1,2,1] )
0.66666666666666663
>>> f1_score( [1,0,1,0], [0,0,1,1], sample_weight=[1,1,4,1] )
0.80000000000000016

Its intended to be used for emphasizing the importance of some samples w.r.t. the others.


Edited to answer the origin of the F-score:

The F-measure was first introduced to evaluate tasks of information extraction at the Fourth Message Understanding Conference (MUC-4) in 1992 by Nancy Chinchor, "MUC-4 Evaluation Metrics", https://www.aclweb.org/anthology/M/M92/M92-1002.pdf . It refers to van Rijsbergen's F-measure, which refers to the paper by N Jardine and van Rijsbergen CJ - "The use of hierarchical clustering in information retrieval."

It is also known by other names such as Sørensen–Dice coefficient, the Sørensen index and Dice's coefficient. This originates from the 1948 paper by Thorvald Julius Sørensen - "A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons."

$\endgroup$
3
  • 2
    $\begingroup$ I do know how it is calculated; I was looking for a reference as to where it came from or it's usage in machine learning literature (papers, journals, conferences, etc.) $\endgroup$ Jun 7, 2017 at 9:50
  • $\begingroup$ @abhi Did you found any reference ? $\endgroup$
    – khan
    Mar 16, 2018 at 18:54
  • $\begingroup$ @khan No not yet unfortunately. $\endgroup$ May 8, 2018 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.