The Scikit-Learn package in Python has two metrics: f1_score and fbeta_score. Each of these has a 'weighted' option, where the classwise F1-scores are multiplied by the "support", i.e. the number of examples in that class.

Is there any existing literature on this metric (papers, publications, etc.)? I can't seem to find any.

  • $\begingroup$ Did you find any reference or how the F1-Score is calculated. $\endgroup$ – khan Mar 16 '18 at 18:55
  • $\begingroup$ Does the update answer your question? $\endgroup$ – Sandeep S. Sandhu Jun 22 '18 at 5:05

The F1 Scores are calculated for each label and then their average is weighted by support - which is the number of true instances for each label. It can result in an F-score that is not between precision and recall.

For example, a simple weighted average is calculated as:

>>> import numpy as np;
>>> from sklearn.metrics import f1_score
>>> np.average( [0,1,1,0 ], weights=[1,1,1,1] )
>>> np.average( [0,1,1,0 ], weights=[1,1,2,1] )
>>> np.average( [0,1,1,0 ], weights=[1,1,4,1] )

The weighted average for each F1 score is calculated the same way:

f_score = np.average(f_score, weights=weights)

For example:

>>> f1_score( [1,0,1,0], [0,0,1,1] )
>>> f1_score( [1,0,1,0], [0,0,1,1], sample_weight=[1,1,2,1] )
>>> f1_score( [1,0,1,0], [0,0,1,1], sample_weight=[1,1,4,1] )

Its intended to be used for emphasizing the importance of some samples w.r.t. the others.

Edited to answer the origin of the F-score:

The F-measure was first introduced to evaluate tasks of information extraction at the Fourth Message Understanding Conference (MUC-4) in 1992 by Nancy Chinchor, "MUC-4 Evaluation Metrics", https://www.aclweb.org/anthology/M/M92/M92-1002.pdf . It refers to van Rijsbergen's F-measure, which refers to the paper by N Jardine and van Rijsbergen CJ - "The use of hierarchical clustering in information retrieval."

It is also known by other names such as Sørensen–Dice coefficient, the Sørensen index and Dice's coefficient. This originates from the 1948 paper by Thorvald Julius Sørensen - "A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons."

  • 2
    $\begingroup$ I do know how it is calculated; I was looking for a reference as to where it came from or it's usage in machine learning literature (papers, journals, conferences, etc.) $\endgroup$ – Abhishek Divekar Jun 7 '17 at 9:50
  • $\begingroup$ @abhi Did you found any reference ? $\endgroup$ – khan Mar 16 '18 at 18:54
  • $\begingroup$ @khan No not yet unfortunately. $\endgroup$ – Abhishek Divekar May 8 '18 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.