Averaging precision and recall when using cross validation

Question

I have performed classification using multiple classifiers for a 2-classes labelled data, and I used 5-fold cross validation. For each fold I calculated tp, tn, fp, and fn. Then I calculated the accuracy, precision, recall and F-score for each test. My question is, when I want to average the results, I took the average of accuracies, but can I average precision, recall and the F-score as well? Or would this be mathematically wrong? P.S. The datasets used in each fold are well balanced in terms of the number of instances per class.

Thanks.

Answer 1

The $F$-score, assuming you're using the usual definition, is already a combination of the precision and recall. Specifically, it is the harmonic mean of them. In other words $$F_1 = 2\cdot\frac{\textrm{precision} \cdot \textrm{recall}}{\textrm{precision} + \textrm{recall}}$$ It's meant to capture the 'effectiveness' of a system where the user places equal weights on precision and recall. There's an extension, called the $F_\beta$ score, which gives $\beta$ times more weight to recall than precision. $$ F_\beta = (1+\beta^2) \frac{\textrm{precision} \cdot \textrm{recall}}{(\beta^2 \cdot\textrm{precision}) + \textrm{recall}} $$ On the other hand, if you're asking whether you can average the 5 $F$ scores (one from each fold), then the answer is yes. In fact, that's the typical way to report a system's performance!

Just be aware that there are some issues with using these values to make inferences about the classifiers' generalization error. For example, a $t$-test between the $F$ scores for one classifier and the $F$ scores for another classifier is going to be too optimistic.