11
$\begingroup$

I have performed classification using multiple classifiers for a 2-classes labelled data, and I used 5-fold cross validation. For each fold I calculated tp, tn, fp, and fn. Then I calculated the accuracy, precision, recall and F-score for each test. My question is, when I want to average the results, I took the average of accuracies, but can I average precision, recall and the F-score as well? Or would this be mathematically wrong? P.S. The datasets used in each fold are well balanced in terms of the number of instances per class.

Thanks.

$\endgroup$
  • 3
    $\begingroup$ I encountered your same problem regarding computing the F-measure (harmonic mean of precision and recall) using cross-validation. In this paper they actually demonstrated that computing the F-measure on the complete set, and not averaging, is the less biased method. I hope this can help $\endgroup$ – markusian Oct 13 '14 at 14:32
  • 3
    $\begingroup$ @markusian Please add this as an answer! It is by far the most important thing on this page!! $\endgroup$ – drevicko Feb 3 '16 at 19:03
2
$\begingroup$

The $F$-score, assuming you're using the usual definition, is already a combination of the precision and recall. Specifically, it is the harmonic mean of them. In other words $$F_1 = 2\cdot\frac{\textrm{precision} \cdot \textrm{recall}}{\textrm{precision} + \textrm{recall}}$$ It's meant to capture the 'effectiveness' of a system where the user places equal weights on precision and recall. There's an extension, called the $F_\beta$ score, which gives $\beta$ times more weight to recall than precision. $$ F_\beta = (1+\beta^2) \frac{\textrm{precision} \cdot \textrm{recall}}{(\beta^2 \cdot\textrm{precision}) + \textrm{recall}} $$ On the other hand, if you're asking whether you can average the 5 $F$ scores (one from each fold), then the answer is yes. In fact, that's the typical way to report a system's performance!

Just be aware that there are some issues with using these values to make inferences about the classifiers' generalization error. For example, a $t$-test between the $F$ scores for one classifier and the $F$ scores for another classifier is going to be too optimistic.

$\endgroup$
  • $\begingroup$ Yes, I used the first formula. This means that averaging the F-score from the different tests yields similar results to averaging precision and recall and then calculating the F-score from them. I tried this on the results I have and it was almost the same. Thanks. $\endgroup$ – Kalaji Aug 8 '13 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.