Questions tagged [rouge]

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.

Filter by
Sorted by
Tagged with
0 votes
1 answer
143 views

ROUGE-N for multiple references

In the original paper (https://aclanthology.org/W04-1013.pdf) that presented ROUGE the description of ROUGE-N is rather unclear on the case when multiple references are used. Firstly it presents the ...
  • 11
0 votes
1 answer
76 views

Why isn't ROUGE-N normalized by the number of N-grams in the reference summary?

Note: I'll focus on $ROUGE-1$, but the same holds for $ROUGE-N$. For a machine-produced summary $M$ and a bunch of reference summaries $RefSummaries$, I believe $ROUGE-1$ can be calculated in the ...
2 votes
1 answer
573 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
  • 71
18 votes
2 answers
23k views

Interpreting ROUGE scores

I recently read the paper on Salesforce's advances in abstractive text summarisation. This states that the ROUGE-1 score achieved of 41.16 is significantly better than the previous state of the art. ...
4 votes
1 answer
1k views

Gaming the ROUGE metric for text summarization

ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated). $$ROUGE_{n}= \frac {\sum_{s\...
  • 370