0
$\begingroup$

So I am using the BLEU score metric to evaluate and compare my neural machine translation model's performance with existing models. However, I'm wondering how many criteria do I have to match with the other models.

Criteria like dev sets, test sets and hyperparameters I think are doable. However, the preprocessing step I use can be different from existing models and so I'm wondering if the BLEU score of my model can be properly compared with others. Do I have to train the model on the same training datasets as the previous models? There are also chances that existing models have hidden parameters that were not reported.

https://arxiv.org/pdf/1804.08771.pdf addresses the problem of reporting BLEU and calls to switch to SacreBLEU. But many existing models use BLEU so I don't think I can use the SacreBLEU score metric on my model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.