Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I have a bunch of experiments in which I am calculating precision and recall. I want to present a mean precision and recall for these experiments. Should these values be weighted by anything?

share|improve this question
In general I would not recommend attempting to define any sort of "mean" across values from different experiments, unless you have a clear idea of what these "means" are supposed to represent. Otherwise you will usually be computing a "statistic" which is useless at best and misleading at worst. – charles.y.zheng Mar 9 '11 at 11:23
I would generally agree but these experiments are very much related. – Raffi Khatchadourian Mar 9 '11 at 16:39
Sorry, but your question is rather general ... can you provide how this experiments are related ? If they originate from different folds of a x-validation, an unweighted average is (of course ;)) meaningful, but in general this question is hard to answer. – steffen Mar 10 '11 at 14:49

1 Answer

up vote 1 down vote

It would be good to know more about the experiments. But there's two main ways (at least in information retrieval) of averaging contingency-table based measures like recall and precision:

  1. Compute the individual measures for each experiment and take the unweighted average ("macroaveraging").

  2. Add up the contingency tables and compute the measures from the summed contingency table ("microaveraging").

The usual reference on these is

Tague, J. The pragmatics of information retrieval experimentation. In Information Retrieval Experiment, Butterworths, London, 1981, pp 59-102.

Yang, Rose, Li, and I also discuss these two approaches, and provide sample data sets in RCV1: A New Benchmark Collection for Text Categorization Research.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.