2
$\begingroup$

I have a system whose performance was characterized in terms of the amount of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). There is a rate that summarizes all this quantities in one, called accuracy, and is defined like so (according to Wikipedia): $$ a=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}. $$ How ever, according to this Microsoft's blog, the F-score is a better choice than accuracy because it works even if I the class distribution is uneven. F-score is the harmonic mean of precision and recall, but it does not include the amount of true negatives (TN): $$ F = \frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}} = \frac{2}{\frac{1}{p}+\frac{1}{r}}. $$ To consider the amount of TN could I calculate the mean of the precision, recall and specificity like below? $$ u = \frac{3}{\frac{1}{p}+\frac{1}{r}+\frac{1}{s}} $$ How is that called? And why it haven't been used before?

Note: this question is not the same as the one discussed here, because I am asking why F-measure does not include specificity (and if I can include it), not why accuracy is not a good metric for an uneven number of positives and negatives.


Trying to answer my own question I see that: $$ u=\frac{(3\text{TN}\,\text{TP})}{(3\text{TN}\,\text{TP})+\text{TN}\,\text{FN}+\text{TN}\,\text{FP}+\text{TP}\,\text{FP}}, $$ which I do not know how to interpret. Perhaps I should start using the Matthews correlation coefficient, given that Wikipedia says it is one of the best measurements to describe the confusion matrix (and "can be used even if the classes are of very different sizes"). After all, I was looking to include TP, TN, FP and FN in a single quantity.

$\endgroup$
1
  • $\begingroup$ I just edited the question to explain why this question is not a duplicate $\endgroup$
    – JMFS
    Commented Mar 15, 2018 at 14:47

2 Answers 2

1
$\begingroup$

Well i think instead of adding precision, recall and specificity, you can only use harmonic mean of specificity and sensitivity, that will help you capture all four TP, TN, FP and FN in a single quantity.
The harmonic mean between specificity and sensitivity referred as F-measure.
And when it comes to interpretation F1score doesn't have any interpretation, please correct me if i am wrong. and i think MCC gives you +1, 0 and -1, which is of no use if you are using it for evaluating a machine learning model while training or testing.

$\endgroup$
1
$\begingroup$

As already mentioned, this is widely used and known as F1.

Now, you are saying:

I am asking why F-measure does not include specificity (and if I can include it), not why accuracy is not a good metric for an uneven number of positives and negatives.

I think you were close to your answer. As for accuracy, specificity is not good under unbalanced classes. For a large N/P ratio, specificity = TN / (TN + FP) ~= 1.0 because TN becomes disproportionate with respect to FP unless you have a very large number of FP with respect to P, i.e., your classifier needs to be atrociously bad for it to be reflected into specificity if N/P is large. Classification cases with large N/P are common, e.g., for uncommon events or diseases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.