How does recall help precision overcome "length-related problems?"

Question

I'm reading the paper on the Bilingual Evaluation Understudy (BLEU) metric BLEU: A Method for Automatic Evaluation of Machine Translation (Papineni et al., 2002) and had a question regarding a quote in the paper.

The quote is motivated by the observation that good translation metrics should also take the length of the translation into account. Until now, the main focus of the paper had been modified n-gram precision, which is unable to properly account for translation candidates that are either too short or too long.

The paper states that:

Traditionally, precision has been paired with recall to overcome such length-related problems. However, BLEU considers multiple reference translations, each of which may use a different word choice to translate the same source word.

I'm having some trouble understanding that statement. My understanding of recall is that it adds false negatives rather than false positives in the denominator. In a more typical situation, I can imagine that a high precision due to low false positives could be counteracted with a low recall due to many false negatives. However, I'm having trouble how we can apply that concept to translation. I'm also having trouble understanding how using multiple reference translations would hinder us from using recall to counteract n-gram precision.

Any tips are appreciated. Thanks.

Answer 1

I think the authors would very much like to include something like recall. After all, when translating into English, a degenerate sentence "the" would score perfectly on precision, but very poor on recall, whatever the exact definition of recall in the context of translation is. You not only want to have all words correct but also cover the entire reference sentence with the translation.

When we have multiple references, computing the precision is simple: if the n-gram is in at least one of the references, it is probably correct. If you perfectly match one of the reference sentences, you probably want to get a total score of 1. Because there is only one such (perfectly recalled) reference, you would probably need to take the maximum-recall sentence.

Now imagine, you have a sentence with two clauses. You have translated the first clause according to one of the references, the second clause according to another one reference. Your precision is 1, you want your recall to be 1 as well. How can you do it? In this way, you could think about more and more examples, where computing recall is tricky.

However, in the paper, they make the following observation: the only way to make the precision too high at the expense of recall is to make the sentence short to avoid outputting risky words. Therefore, they come to the conclusion that they can only penalize sentences that are too short. In order not to be penalized, the model needs to output sentences that are long enough, but if they are incorrect, they get penalized by the precision.