0
$\begingroup$

I looked online and am unable to find the proper classification metric for machine translation problems.

Let's the predicted output is: "My name John" and the ground truth is "My name is John"

I know I can compute word error rate (which is computed using edit distance). And if the number of words are the same, then classification error is simply equal to word error rate. Then classification accuracy is simply 1 minus classification accuracy.

But how do I compute classification error when the number of words are different like in my above example?

$\endgroup$
1
  • $\begingroup$ There are many standard metrics for evaluating machine translation systems. Start by searching for BLEU and ROUGE. $\endgroup$
    – tddevlin
    Jun 2 '20 at 1:06
1
$\begingroup$

Machine translation cannot be easily evaluated by measuring classification accuracy. For every source sentence, there are very many valid translations, but your reference sentence is only one of the many possible correct ones.

Evaluation of machine translation is still an open problem. There is even an annual competition where teams try to develop a metric that correlates best with human assessments. The current most popular metrics are BLEU score based on n-gram precision and METEOR score based on monolingual alignment and F-measure. Recently, also metrics based on BERT seem to perform very well. All these metrics operate on the output string and are independent of the translation model.

With sequence-to-sequence models that condition each output symbol on the previous one (i.e., autoregressive models), it might make sense to measure the accuracy given the prefix from the reference sentence, i.e., you have a prefix

My name is

and you want to predict John. In such a setup, you still have a problem that the model might prefer a different wording and the accuracy would unfair to the model, but in most contexts, there is the only option that is obviously true, so you get a reasonable estimate of the accuracy.

Now, the problem is that unlike the previously mentioned metrics, the accuracy depends on how you segment the data. You could for instance artificially boost the accuracy by using small subword units, without actually improving the translation quality. Also, there are experimental models that are no autoregressive and generate the output partially in parallel in random order. In that case, measuring the accuracy does not make really sense.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.