So I am using the BLEU score metric to evaluate and compare my neural machine translation model's performance with existing models. However, I'm wondering how many criteria do I have to match with the other models.

Criteria like dev sets, test sets and hyperparameters I think are doable. However, the preprocessing step I use can be different from existing models and so I'm wondering if the BLEU score of my model can be properly compared with others. Do I have to train the model on the same training datasets as the previous models? There are also chances that existing models have hidden parameters that were not reported.

https://arxiv.org/pdf/1804.08771.pdf addresses the problem of reporting BLEU and calls to switch to SacreBLEU. But many existing models use BLEU so I don't think I can use the SacreBLEU score metric on my model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.