Questions tagged [rouge]

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.

Filter by
Sorted by
Tagged with
0
votes
0answers
36 views

Why isn't ROUGE-N normalized by the number of N-grams in the reference summary?

Note: I'll focus on $ROUGE-1$, but the same holds for $ROUGE-N$. For a machine-produced summary $M$ and a bunch of reference summaries $RefSummaries$, I believe $ROUGE-1$ can be calculated in the ...
2
votes
1answer
150 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
15
votes
2answers
18k views

Interpreting ROUGE scores

I recently read the paper on Salesforce's advances in abstractive text summarisation. This states that the ROUGE-1 score achieved of 41.16 is significantly better than the previous state of the art. ...
4
votes
1answer
845 views

Gaming the ROUGE metric for text summarization

ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated). $$ROUGE_{n}= \frac {\sum_{s\...