0
$\begingroup$

I have run an experiment where users were given a distorted sentence and asked to restore it. Currently, I am interested to grade the results on the scale between 0 to 1. I'm wondering if there any well-known ways to do so?

I am not interested in the "semantic quality" but only in the word for word similarity.

$\endgroup$
1
$\begingroup$

There is the BLEU score (see https://en.wikipedia.org/wiki/BLEU ) used in machine translation that might fit your intentions.

For a more detailed discussion of the BLEU score see this stackoverflow question: https://stackoverflow.com/questions/5390397/bleu-score-implementation-for-sentence-similarity-detection

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ From the WikiPedia article: "BLEU is designed to approximate human judgement at a corpus level, and performs badly if used to evaluate the quality of individual sentences." $\endgroup$ – fnl Jul 27 '15 at 13:20
  • $\begingroup$ @fnl I thought it might be applicable because the OP states "I am not interested in the "semantic quality" but only in the word for word similarity." $\endgroup$ – user79309 Jul 27 '15 at 13:23
1
$\begingroup$

There are many solutions for this kind of problem. Have you tried for instance the Levenshtein distance, jaccard index overlap coefficient, or any other string distance method? Wikipedia link

Keep in mind that your sentences should be tokenized into words before applying these methods...

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.