How to aggregate evaluation metrics from many evaluations? Say I'm developing an automatic term extractor and I have a lot of documents. Let's assume this documents do not all have the same size. I also have a list of terms that should be extracted from each document, the gold standards. 
When I apply the term extractor to each of the documents I get a list of terms and I can calculate, for example, the precision score of the extractor for each document. For example:  
Document 1


*

*Gold -> ('head', 'nail', 'toe')

*Extracted -> ('head', 'cereal')

*Precision = 0.5


Document 2 


*

*Gold -> ('belly', 'limb', 'migraine', 'colon')

*Extracted -> ('limb', 'tea', 'John', 'red')

*Precision = 0.25


Ok, now I have a precision score of my term extractor for each document, so I have a lot of precision scores. How can I aggregate them into just one precision score which can be used to evaluate the system, in the case that:


*

*I only have this precision values?

*I have the raw data?

 A: You can view the task as multi-label classification, and use the typical metrics for multi-label classification, e.g. excerpt from {2}:

Wikipedia also has a section on Statistics and evaluation metrics for Multi-label classification.
But keep in mind that term extraction is a bit different, as explained in {1}:

Despite these first evaluation experiences, no comprehensive
  and global framework has yet been proposed
  for computational terminology as there exist for many
  other NLP fields. Beside economic factors, it seems
  that evaluating terminology acquisition raises some
  specific intrinsic difficulties.
[...]
As mentioned above, traditional metrics of precision
  and recall are not appropriate for term extraction evaluation.
  One problem is that term relevance is a gradual
  rather than a binary notion and that one cannot
  expect all extractors or terminologists to deliver
  ranked list of terms. This led us to stem relevance on
  a terminological distance. A second problem is that
  no terminological standard can be considered as a stable
  and unique gold standard.

FYI Getting the accuracy for multi-label prediction in scikit-learn

References:


*

*{1} Nazarenko, Adeline, and Haifa Zargayouna. "Evaluating term extraction." In International Conference Recent Advances in Natural Language Processing (RANLP'09), pp. 299-304. 2009. https://scholar.google.com/scholar?cluster=10075060291670172889&hl=en&as_sdt=0,22 ; http://anthology.aclweb.org/R/R09/R09-1.pdf#page=323 ; https://pdfs.semanticscholar.org/a953/001b2be9b73574418ba42f8f48017629e11e.pdf

*{2} Tsoumakas, Grigorios, and Ioannis Katakis. "Multi-label classification: An overview." Dept. of Informatics, Aristotle University of Thessaloniki, Greece (2006).

