In the original paper (https://aclanthology.org/W04-1013.pdf) that presented ROUGE the description of ROUGE-N is rather unclear on the case when multiple references are used.

Firstly it presents the ROUGE-N as $$ \text{ROUGE-N} = \frac{\sum_{S \in \text{references}} \sum_{gram_n \in S} \text{Count}_{match}(\text{gram}_n)}{\sum_{S \in \text{references}} \sum_{gram_n \in S} \text{Count}(\text{gram}_n)} $$ and defines the $\text{Count}_{match}$ as the maximum number of n-grams co-occurring in a candidate summary and a set of reference summaries. So it seems like it is meant among the all references and not pairwise.

On the other hand the author starts next section (2.1 Multiple References) with: So far, we only demonstrated how to compute ROUGE-N using a single reference.

So the author acts like he was not talking about multiple references before, but the formula is certainly using them.

Then the case for multiple references is described as that, we compute pairwise summary level ROUGE-N between a candidate summary $s$ and every reference, $r_i$, in the reference set. We then take the maximum of pairwise summary-level ROUGE-N scores.

Paper then describes this process with this formula $$ \text{ROUGE-N}_{\text{multi}} = argmax_i \text{ROUGE-N}(r_i,s) $$

But these descriptions and formulas are inconsistent. I believe that the formulas should be rather $$ \text{ROUGE-N}(r,s) = \frac{\sum_{gram_n \in r} \text{Count}_{match}(\text{gram}_n, r, s)}{\sum_{gram_n \in r} \text{Count}(\text{gram}_n, r)} $$

$$ \text{ROUGE-N}_{\text{multi}} = max_{r \in R} \text{ROUGE-N}(r,s) $$ Where the $\text{Count}_{match}$ is just for the single reference and $R$ is a set of references. Am I right?


2 Answers 2


I have looked into ROUGE-N implementations and found that there are three ways to compute ROUGE:

  • (a) compute a ROUGE for each reference summary and then take their maximum value [1,2,3,4]
  • (b) compute the micro average of ROUGEs, which is the first definition in the question [4]
  • (c) compute the macro average of ROUGEs [2,4]

I think the possible interpretation is (a) or (b). Although (b) seems to be the original definition, (a) and (c) are prevalent in the implementations.


[1] https://github.com/google-research/google-research/tree/master/rouge implementation by google research

[2] https://torchmetrics.readthedocs.io/en/stable/text/rouge_score.html pytorch wrapper of [1]

[3] https://huggingface.co/spaces/evaluate-metric/rouge huggingface wrapper of [1]

[4] https://github.com/li-plus/rouge-metric/blob/master/rouge_metric/RELEASE-1.5.5/ROUGE-1.5.5.pl

  • $\begingroup$ Importantly, it should be noted that the original script (reference [4] in the above answer) actually provides an option to decide which interpretation is chosen! See this line and subsequent implications, such as in the ROUGE-N calculation. $\endgroup$
    – dennlinger
    Apr 14, 2023 at 9:31

I was reading the ROUGE paper right now and I came across with the same doubt, so I was hoping to find someone else with the same question.

I agree with you: in the first formula the summation over the reference set should be removed, otherwise it makes no sense how the author starts the next section, as you pointed out.

I'm a bit surprised by the fact that nobody else seems to have the same doubt despite the importance of the ROUGE metric, so I hope someone else shares their opinion.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.