Which metric to use for language translation?

So I am using a pre-trained model to do the language translation

Eg:

Input  = "Good morning"

Output = "Bonjour"


I would like to see if the translation is correct. I do not have any reference sentence for the translated output.

Can someone please tell me how can I compare the input and output and know if the translation is right.

Basically my project is on language translation. If I pass a sentence (any indian language) it should translate the sentence into a indian language I mention so how can I compare them.

Thankyou!

• Sorry, I deleted my comment. Here it is again: In order to assess the accuracy of an estimated translation, you will need a reference sentence for the output. May 17 at 17:43
• Compare the input/output for what? May 17 at 17:43
• @Galen So is that the only way to do it. Can you suggest some way to evaluate the project I am doing If I pass a sentence (any indian language) it should translate the sentence into a indian language I mention so how can I compare them. May 17 at 17:44
• You can only know if the output is correct by having a reference sentence for the output. Comparing the input and output for the accuracy of the output won't work. May 17 at 17:46
• If I presented myself to you and claimed that a god granted me the ability to translate any English sentence into the language spoken by the master race on the planet Epsilon Eridani 4, how would you verify that I'm not just spouting gibberish?
– whuber
May 17 at 17:52

The task you are looking for is called machine translation quality estimation or sometimes reference-free evaluation (there is a nuanced difference in evaluation, cf. Results of the WMT21 Metrics Shared Task). You can search for tools for that.

If you work with languages that appear in the annual WMT competition, the best choice is the COMET score (paper). It is based on the XLM-R model, so it might work reasonably well also for other languages. Alternatively, BERTScore (paper) based on mBERT or XLM-R might also get reasonable results.

As per my comment above:

You can only know if the output is correct by having a reference sentence for the output. Comparing the input and output for the accuracy of the output won't work.

Sanity checks from partial information

But let us try to make something out of your question by

• taking the absence of a reference sentence as a given constraint, and
• supposing that evidence of correctness doesn't have to be a complete and certain verification of correctness.

If you have some partial information about the language, you might use that to perform sanity checks that the output is reasonable.

Example

Let's consider an example of such a sanity check. Suppose you have a textfile of all words in the output language, including whatever variants that consider conjugations, modified syntax, and slang. You could check that each word in the output of your model is a word according to your word list. If you found that the output contained words that are not in your word list, you might wonder if it is really correct. The approach would be heavily dependent on the completeness of your list of words, and would at-best give some limited evidence of correctness/incorrectness.

Is your output necessarily correct just because it only contains known words of the language? No. This word check doesn't tell you about whether the syntax/grammar or semantics are correct (e.g. cat; the:, desk: cow, brief, multiplication).

So take sanity checks as both tentative and partial assessments of the correctness. You can check if something quacks like a duck, but you won't know what it is really saying.

If you know nothing about the output language, then there is nothing to check.