calculating recall/precision from k-fold cross validation (or leave-one-out) can be performed either by averaging the recall/precision values obtained from different k folds or by combining the predictions and then calculate one value for each of the recall and precision.

My data is composed of 1000 samples belonging to two classes. the positives are only 50 while negatives are 950 samples. If we apply leave-one-out using the averaged k-fold cross validation approach. Then, we will notice that we have the precision and recall in 950 folds are not defined (NaN) because NP/(TP+FP) are all zeros (as only one negative sample is examined in this k-fold). I think, such NaN might be replaced by 1 because no meaning to set its value to be 0 (because the current sample is negative and it was correctly predicted to be negative). Then, we average the recall/precision from 1000-folds. the values will be different than obtaining the all the predictions and then calculating one precision/recall.

The same difference will happen when we apply k-fold cross validation on this data instead of leave-one-out (even if we produce balanced ratio of positives in each fold).

So, my question is: which approach is more accurate to calculate the precision/recall from k-fold cross validation (combined or average)? is the same situation for leave-one-out?

There are two relevant discussions : Mean(scores) vs Score(concatenation) in cross validation and Averaging precision and recall when using cross validation

However, the first one did not have clear answer for this point and the second discusses the averaging effect of Fmeasure (not in the precision/recall). both discussions did not discuss if leave-one-out has other preference.

  • $\begingroup$ Perhaps this is too pedantic, but to be clear, leave-one-out IS k-fold cross validation. It's a special case, where k=n. $\endgroup$ – Jean V. Adams Aug 20 '13 at 15:10
  • $\begingroup$ I encountered your same problem regarding computing the F-measure (harmonic mean of precision and recall) using cross-validation. In this paper they actually demonstrated that computing the F-measure on the complete set, and not averaging, is the less biased method. I hope this can help $\endgroup$ – markusian Oct 13 '14 at 14:33
  • $\begingroup$ Indeed!, as you said, I found that calculating F-measure on complete set is less biased and I did that in my project which I finished some time ago!. $\endgroup$ – Abbas Oct 13 '14 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.