The Scikit-Learn package in Python has two metrics: f1_score and fbeta_score. Each of these has a 'weighted' option, where the classwise F1-scores are multiplied by the "support", i.e. the number of examples in that class.
Is there any existing literature on this metric (papers, publications, etc.)? I can't seem to find any.
The F1 Scores are calculated for each label and then their average is weighted by support - which is the number of true instances for each label. It can result in an F-score that is not between precision and recall.
For example, a simple weighted average is calculated as:
>>> import numpy as np;
>>> from sklearn.metrics import f1_score
>>> np.average( [0,1,1,0 ], weights=[1,1,1,1] )
0.5
>>> np.average( [0,1,1,0 ], weights=[1,1,2,1] )
0.59999999999999998
>>> np.average( [0,1,1,0 ], weights=[1,1,4,1] )
0.7142857142857143
The weighted average for each F1 score is calculated the same way:
f_score = np.average(f_score, weights=weights)
For example:
>>> f1_score( [1,0,1,0], [0,0,1,1] )
0.5
>>> f1_score( [1,0,1,0], [0,0,1,1], sample_weight=[1,1,2,1] )
0.66666666666666663
>>> f1_score( [1,0,1,0], [0,0,1,1], sample_weight=[1,1,4,1] )
0.80000000000000016
Its intended to be used for emphasizing the importance of some samples w.r.t. the others.
Edited to answer the origin of the F-score:
The F-measure was first introduced to evaluate tasks of information extraction at the Fourth Message Understanding Conference (MUC-4) in 1992 by Nancy Chinchor, "MUC-4 Evaluation Metrics", https://www.aclweb.org/anthology/M/M92/M92-1002.pdf . It refers to van Rijsbergen's F-measure, which refers to the paper by N Jardine and van Rijsbergen CJ - "The use of hierarchical clustering in information retrieval."
It is also known by other names such as Sørensen–Dice coefficient, the Sørensen index and Dice's coefficient. This originates from the 1948 paper by Thorvald Julius Sørensen - "A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons."
