How to compare two texts with different order of words?

Question

I have two texts, one ground truth and one OCR result, and I want to measure to what accuracy the result matches the ground truth. But since the text source is non-linear, both texts have a different order of words (imagine an invoice and two people would read it in a different order), so I can't compare the texts word by word.

How do I best compare these two texts? I currently have this approaches:

Measure the occurence of each character (or better: word) in ground truth and result and compare the numbers. The closer both numbers are, the higher the accuracy. In the end, I take the mean accuracy. But here I have no idea how to weigh the results, since some characters/words occur more often than others and should therefore count more.

Another issue is how to deal with characters/words that only occure in one of the texts. How do I penalize this?

Answer 1

Please use the cosine similarity of the tf-idf matrix.

tf-idf matrix is used to quantify words in your documents, and cosine similarity measures the similarity among your documents. It only accounts the frequency of words in the documents, not the order. The typical range of the cosine similarity matrix is [-1,1], but in document similarity, there is no negative value, which indicates that the range would be [0,1].

For penalty issues for low occurrence, you can handle them on the idf part from the equation.