11
$\begingroup$

I understand that f-measure (based on precision and recall) is an estimate of how accurate a classifier is. Also, f-measure is favored over accuracy when we have an unbalanced dataset. I have a simple question (which is more about using correct terminology than about technology). I have an unbalanced dataset and I use f-measure in my experiments. I am about to write a paper which is NOT for a machine learning/ data mining conference. Hence, can I refer to f-measure synonymously with accuracy in this context. For eg, I have a f-measure of 0.82, then can I say my classifier achieves 82% accurate predictions?

$\endgroup$
  • $\begingroup$ It would be better to introduce the f-measure if you use it. Substituting the two is not correct in my point of view. In your case if your accuracy is 99% you'll achieve 99% accurate predictions not matter what your f-measure is, and it could lead readers into error. $\endgroup$ – AdrienNK May 24 '14 at 12:01
  • $\begingroup$ @AdrienNK: 99% accuracy does not imply 99% correct predictions unless the relative frequencies of the test cases are the same as in the real application situation. $\endgroup$ – cbeleites unhappy with SX May 24 '14 at 14:41
  • $\begingroup$ @cbeleites you're right, I know, but often the test cases are issued from the same distribution (well maybe that's the biased view I have of it because I rarely had to work with data on which that wasn't the case) $\endgroup$ – AdrienNK May 24 '14 at 15:01
  • $\begingroup$ @AdrienNK: I'm analytical chemist working towards medical diagnoses. The prevalence of the disease in question may vary about orders of magnitude between different patient subpopulations. See e.g. the discussion of the different PPVs in the second half of this article: nature.com/news/2011/110323/full/471428a.html $\endgroup$ – cbeleites unhappy with SX May 24 '14 at 15:09
  • 2
    $\begingroup$ That was a fascinating read, thank you for bringing that to my attention. $\endgroup$ – AdrienNK May 24 '14 at 17:15
13
$\begingroup$

First of all, I find "accuracy" sometimes a bit misleading, as it refers to distinct things:

The term accuracy in geneal for evaluating systems or methods (I'm analytical chemist) refers to the bias of predictions, i.e. it answers the question how good predictions are on average.

As you know, there are lots of different performance measures that answer different aspects of performance for classifiers. One of them happens to be called accuracy as well. If your paper is not for a machine learning/classification audience, I'recommend to make this distinction very clear. Even for this more specific meaning of accuracy I'd be very explicit of what I call accuracy as again several ways of dealing with class imbalance may occur. Typically, class imbalance is ignored, leading to the well-known $\frac{TP+TN}{all~cases}$ calculation. However, you may also use the average of sensitivity and specificity, which amounts to controlling class imbalance by weighting your average.

The F-score is often introduced as harmonic mean of precision and recall (or positive predictive value and sensitivity). For your question I think it helpful to spell this out a bit further and simplify it:

$F = \frac{2 \cdot precision \cdot recall}{precision + recall} = \frac{2 \frac{TP}{all~P} \frac{TP}{all T}}{\frac{TP}{all~P} + \frac{TP}{all T}} = \frac{2 \frac{TP^2}{all~P \cdot all T}}{\frac{TP \cdot all~T}{all~P \cdot all T} + \frac{TP \cdot all~P}{all~P \cdot all T}} = \frac{2~TP^2}{TP \cdot all~T + TP \cdot all~P} = \frac{2~TP}{all~T + all~P} $

The last expression is not a fraction of anything that I can think of as a certain group of test cases. In particular, a (heavy) overlap between the TRUE and the POSITIVE cases is expected. This would keep me from expressing an F-score as percentage as that kind of implies a proportion of cases. Actually, I think I'd warn the reader that F-score does not have such an interpretation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ more specifically this is $F_1$ measure. F-score can be generalized with a separate parameter $\endgroup$ – qwr Dec 9 '18 at 7:19
0
$\begingroup$

Quick Answer:

No, F-measure formula does not consist of TN factor, and it is useful on retrieve problems (doc).

Thus, it's (F-measure) the correct approach to evaluate the imbalanced datasets or in retrieval problems case instead of accuracy and ROC.

Accuracy = (TP+TN) / (TP+FP+FN+TN)

F1_Score = 2*(Recall * Precision) / (Recall + Precision)
# or
F1_Score = 2*TP / (2*TP + FP + FN)

[NOTE]:

Precision = TP / (TP+FP)

Recall = TP / (TP+FN)
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.