1
$\begingroup$

For machine translation purposes I use bleu score, which seems to be the validation mechanism of choice (used in the sutskever 2014 sequence-to-sequence).

The purpose is to get as high bleu as possible (between 0 to 1).

The following mumble gives an extraordinary high bleu score (0.77):

from nltk import bleu_score

reference = ['The moon is very bright']
hypothesis = ['Dee dd ss eee']
reference = [[r.split()] for r in reference]
hypothesis = [[h.split()] for h in hypothesis]

bleu_score.corpus_bleu(reference, hypothesis)

Why does the bleu score give such high accuracy for mumbling? Which other tools for validation could I use for machine translation?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Which other tools for validation could I use for machine translation?

NLTK's BLEU implementation has a few issues. One of the most commonly used BLEU implementation is the one provided with MOSES (documentation).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Commenting for posterity: while this is true, the high BLEU score here was user error. (The other answer addresses this.) Following Franck's guidance by using the Moses implementation of BLEU (or the more recent SacreBLEU) would have made the user error obvious. $\endgroup$
    – Arya McCarthy
    Apr 25, 2021 at 4:29
1
$\begingroup$

You are calling the score function incorrectly. This is the way you do it:

from nltk import bleu_score
references = ['The moon is very bright'.split()]
hypothesis = 'Dee dd ss eee'.split()
bleu_score.sentence_bleu(references, hypothesis)

It will print 0 as expected.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.