Comparing corpus complexities

Question

I would like to compare how complex (varied or predictable) are my three corpora. They are from different topics, so some vocabulary is different, some are the same. Looking at one of the data sets, it's clear that the syntax is more difficult than in the other two, sentences are longer, etc.

I built word N-Gram language models using the SRILM toolkit with the idea that I can then compare these models. One measure mentioned in relation to language models is perplexity.

Can I just use perplexities of the three LMs directly as a measure of how varied are the corpora? The vocabulary and the sizes of the corpora are different, so now I think that this won't be a good comparison. I also built LMs from POS-Tags but the quality of the POS-Tagging result is not good because the language is from fora, has spelling mistakes, ungrammatical sentences and so on. What measures could be used to compare complexity of corpora from different domains?

Answer 1

Perplexity is a good measure of the predictability of your text corpora. For example, if the perplexity of your text is 80, then that means that predicting the next word in a sentence after reading the previous words is as hard as a multiple choice exam with 80 choices per question.

Comparing perplexities across corpora works best when you use the same vocabulary everywhere. You can specify a vocabulary when you build the language models in SRILM. You might want to use the same vocabulary for all of the files.