# What metric should I use for evaluating a reading order of text tokens given the correct ordering?

Given an ordering of tokens extracted from a document with a ground truth ordering available. What would be the correct way to evaluate the ordering?

I took a look at some Machine Translation evaluation metrics such as Word Error Rate and the BLEU metric, and these might score highly for the correct ordering, but I wonder if insertions and deletions as operations in WER make sense if I am only interested in the evaluating the ordering (not whether a word was extracted correctly for example)

The BLEU metric relies on n-gram precision and would probably not work well if the document repeats words or n-grams a lot. I think it's designed to work well on short sequences, so if a whole paragraph is out of order (which I expect will be the more common case) it won't work well.

What other metrics would you suggest?

BLEU to some extent captures the word order well. You can get a better idea by trying BLEU-$$n$$ metric, where $$n$$ means the longest $$n$$-gram being consider. (In the early days of image captioning, the typical evaluation metric was BLEU-1). But as you noted, it becomes less efficient with the growing length of the text. But if you could reasonably sentence-split the documents, it should work well.