3
$\begingroup$

I have a bunch of experiments in which I am calculating precision and recall. I want to present a mean precision and recall for these experiments. Should these values be weighted by anything?

$\endgroup$
3
  • $\begingroup$ In general I would not recommend attempting to define any sort of "mean" across values from different experiments, unless you have a clear idea of what these "means" are supposed to represent. Otherwise you will usually be computing a "statistic" which is useless at best and misleading at worst. $\endgroup$ Mar 9, 2011 at 11:23
  • $\begingroup$ I would generally agree but these experiments are very much related. $\endgroup$ Mar 9, 2011 at 16:39
  • $\begingroup$ Sorry, but your question is rather general ... can you provide how this experiments are related ? If they originate from different folds of a x-validation, an unweighted average is (of course ;)) meaningful, but in general this question is hard to answer. $\endgroup$
    – mlwida
    Mar 10, 2011 at 14:49

1 Answer 1

2
$\begingroup$

It would be good to know more about the experiments. But there's two main ways (at least in information retrieval) of averaging contingency-table based measures like recall and precision:

  1. Compute the individual measures for each experiment and take the unweighted average ("macroaveraging").

  2. Add up the contingency tables and compute the measures from the summed contingency table ("microaveraging").

The usual reference on these is

Tague, J. The pragmatics of information retrieval experimentation. In Information Retrieval Experiment, Butterworths, London, 1981, pp 59-102.

Yang, Rose, Li, and I also discuss these two approaches, and provide sample data sets in RCV1: A New Benchmark Collection for Text Categorization Research.

$\endgroup$
5
  • $\begingroup$ Note that if you calculating F-measures, you should use microaveraging - macro averaged F-measures can be misleading. I can dig up a paper investigating this if you're interested. $\endgroup$
    – drevicko
    Sep 17, 2016 at 15:50
  • 1
    $\begingroup$ I'd be interested in the particular reference you mention, but either or both of microaveraging and macroaveraging can be misleading. It all depends on what property of the system you're trying to understand. We discuss this a bit in the RCV1 paper cited above. $\endgroup$ Sep 18, 2016 at 12:47
  • $\begingroup$ "Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement", George Forman, Martin Scholz, dl.acm.org/citation.cfm?id=1882479, ACM SIGKDD Explorations Newsletter, 2010 $\endgroup$
    – drevicko
    Sep 20, 2016 at 11:07
  • $\begingroup$ The Forman paper isn't discuss averaging across multiple classification tasks, which I believe is what the original poster was interested in. It's talking about averaging across partitions in cross-validation: that's a much trickier issue. $\endgroup$ Sep 22, 2016 at 9:16
  • $\begingroup$ I'd almost argue that averaging across multiple tasks is a trickier issue.. Yes, you're right, I realised it was different when I looked it up, but it does illustrate some potential pitfalls. In any case, I'll go read you paper when I've a moment - sounds interesting (: $\endgroup$
    – drevicko
    Sep 23, 2016 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.